Asymptotic theory of in-context learning by linear attention

Asymptotic theory of in-context learning by linear attention

20 May 2024

Jacob A. Zavatone-Veth

Papers citing "Asymptotic theory of in-context learning by linear attention"

10 / 10 papers shown

Title
Training Dynamics of In-Context Learning in Linear Attention Yedi Zhang Aaditya K. Singh Peter E. Latham Andrew Saxe MLT 64 1 0 28 Jan 2025
Pretrained transformer efficiently learns low-dimensional target functions in-context Kazusato Oko Yujin Song Taiji Suzuki Denny Wu 39 4 0 04 Nov 2024
Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokens Vittorio Erba Emanuele Troiani Luca Biggio Antoine Maillard Lenka Zdeborová 23 0 0 24 Oct 2024
Context-Scaling versus Task-Scaling in In-Context Learning Amirhesam Abedsoltan Adityanarayanan Radhakrishnan Jingfeng Wu M. Belkin ReLM LRM 40 3 0 16 Oct 2024
Risk and cross validation in ridge regression with correlated samples Alexander B. Atanasov Jacob A. Zavatone-Veth C. Pehlevan 27 4 0 08 Aug 2024
Spin glass model of in-context learning Yuhao Li Ruoran Bai Haiping Huang LRM 42 0 0 05 Aug 2024
Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond Yingcong Li A. S. Rawat Samet Oymak 25 6 0 13 Jul 2024
The Transient Nature of Emergent In-Context Learning in Transformers Aaditya K. Singh Stephanie C. Y. Chan Ted Moskovitz Erin Grant Andrew M. Saxe Felix Hill 67 32 0 14 Nov 2023
How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? Jingfeng Wu Difan Zou Zixiang Chen Vladimir Braverman Quanquan Gu Peter L. Bartlett 123 49 0 12 Oct 2023
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 250 460 0 24 Sep 2022