v1v2 (latest)

Towards Understanding the Universality of Transformers for Next-Token Prediction

3 October 2024

Papers citing "Towards Understanding the Universality of Transformers for Next-Token Prediction"

26 / 26 papers shown

Title
Transformers are Universal In-context Learners Takashi Furuya Maarten V. de Hoop Gabriel Peyré 94 11 0 02 Aug 2024
Transformers are Expressive, But Are They Expressive Enough for Regression? Swaroop Nath H. Khadilkar Pushpak Bhattacharyya 49 3 0 23 Feb 2024
How do Transformers perform In-Context Autoregressive Learning? Michael E. Sander Raja Giryes Taiji Suzuki Mathieu Blondel Gabriel Peyré 76 10 0 08 Feb 2024
Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape Juno Kim Taiji Suzuki 94 24 0 02 Feb 2024
Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context Xiang Cheng Yuxin Chen S. Sra 68 41 0 11 Dec 2023
Mistral 7B Albert Q. Jiang Alexandre Sablayrolles A. Mensch Chris Bamford Devendra Singh Chaplot ... Teven Le Scao Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LRM 110 2,246 0 10 Oct 2023
Uncovering mesa-optimization algorithms in Transformers J. Oswald Eyvind Niklasson Maximilian Schlegel Seijin Kobayashi Nicolas Zucchet ... Mark Sandler Blaise Agüera y Arcas Max Vladymyrov Razvan Pascanu João Sacramento 72 64 0 11 Sep 2023
Implicit regularization of deep residual networks towards neural ODEs Pierre Marion Yu-Han Wu Michael E. Sander Gérard Biau 93 17 0 03 Sep 2023
One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention Arvind V. Mahankali Tatsunori B. Hashimoto Tengyu Ma MLT 80 102 0 07 Jul 2023
Trained Transformers Learn Linear Models In-Context Ruiqi Zhang Spencer Frei Peter L. Bartlett 93 201 0 16 Jun 2023
Transformers learn to implement preconditioned gradient descent for in-context learning Kwangjun Ahn Xiang Cheng Hadi Daneshmand S. Sra ODL 90 176 0 01 Jun 2023
The emergence of clusters in self-attention dynamics Borjan Geshkovski Cyril Letrouit Yury Polyanskiy Philippe Rigollet 87 56 0 09 May 2023
Transformers as Algorithms: Generalization and Stability in In-context Learning Yingcong Li M. E. Ildiz Dimitris Papailiopoulos Samet Oymak 94 174 0 17 Jan 2023
Transformers learn in-context by gradient descent J. Oswald Eyvind Niklasson E. Randazzo João Sacramento A. Mordvintsev A. Zhmoginov Max Vladymyrov MLT 116 496 0 15 Dec 2022
What learning algorithm is in-context learning? Investigations with linear models Ekin Akyürek Dale Schuurmans Jacob Andreas Tengyu Ma Denny Zhou 119 493 0 28 Nov 2022
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes Shivam Garg Dimitris Tsipras Percy Liang Gregory Valiant 143 514 0 01 Aug 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 208 1,987 0 29 Mar 2022
Sinkformers: Transformers with Doubly Stochastic Attention Michael E. Sander Pierre Ablin Mathieu Blondel Gabriel Peyré 83 85 0 22 Oct 2021
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos Apoorv Vyas Nikolaos Pappas Franccois Fleuret 209 1,793 0 29 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 892 42,463 0 28 May 2020
Are Transformers universal approximators of sequence-to-sequence functions? Chulhee Yun Srinadh Bhojanapalli A. S. Rawat Sashank J. Reddi Sanjiv Kumar 126 359 0 20 Dec 2019
Are Sixteen Heads Really Better than One? Paul Michel Omer Levy Graham Neubig MoE 109 1,069 0 25 May 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,229 0 11 Oct 2018
Neural Ordinary Differential Equations T. Chen Yulia Rubanova J. Bettencourt David Duvenaud AI4CE 448 5,168 0 19 Jun 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 805 132,725 0 12 Jun 2017
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.3K 194,510 0 10 Dec 2015