JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and
Attention

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention

1 October 2023

Yuandong Tian

Zhenyu (Allen) Zhang

Papers citing "JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention"

12 / 12 papers shown

Title
How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias Ruiquan Huang Yingbin Liang Jing Yang 46 0 0 02 May 2025
On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery Renpu Liu Ruida Zhou Cong Shen Jing Yang 28 0 0 17 Oct 2024
Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent Bo Chen Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song 96 19 0 15 Oct 2024
Understanding and Minimising Outlier Features in Neural Network Training Bobby He Lorenzo Noci Daniele Paliotta Imanol Schlag Thomas Hofmann 36 3 0 29 May 2024
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection Jiawei Zhao Zhenyu (Allen) Zhang Beidi Chen Zhangyang Wang A. Anandkumar Yuandong Tian 43 175 0 06 Mar 2024
Linear Transformers are Versatile In-Context Learners Max Vladymyrov J. Oswald Mark Sandler Rong Ge 34 13 0 21 Feb 2024
Implicit Bias and Fast Convergence Rates for Self-attention Bhavya Vasudeva Puneesh Deora Christos Thrampoulidis 34 13 0 08 Feb 2024
An Information-Theoretic Analysis of In-Context Learning Hong Jun Jeon Jason D. Lee Qi Lei Benjamin Van Roy 27 18 0 28 Jan 2024
The Expressibility of Polynomial based Attention Scheme Zhao-quan Song Guangyi Xu Junze Yin 32 5 0 30 Oct 2023
Do Transformers Parse while Predicting the Masked Word? Haoyu Zhao A. Panigrahi Rong Ge Sanjeev Arora 76 31 0 14 Mar 2023
How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding Yuchen Li Yuan-Fang Li Andrej Risteski 120 61 0 07 Mar 2023
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 250 460 0 24 Sep 2022