Do pretrained Transformers Learn In-Context by Gradient Descent?

Do pretrained Transformers Learn In-Context by Gradient Descent?

12 October 2023

Lingfeng Shen

Daniel Khashabi

Papers citing "Do pretrained Transformers Learn In-Context by Gradient Descent?"

6 / 6 papers shown

Title
ICL CIPHERS: Quantifying "Learning'' in In-Context Learning via Substitution Ciphers Zhouxiang Fang Aayush Mishra Muhan Gao Anqi Liu Daniel Khashabi 44 0 0 28 Apr 2025
The Expressibility of Polynomial based Attention Scheme Zhao-quan Song Guangyi Xu Junze Yin 32 5 0 30 Oct 2023
The Learnability of In-Context Learning Noam Wies Yoav Levine Amnon Shashua 122 91 0 14 Mar 2023
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 250 458 0 24 Sep 2022
Large Language Models are Zero-Shot Reasoners Takeshi Kojima S. Gu Machel Reid Yutaka Matsuo Yusuke Iwasawa ReLM LRM 310 4,077 0 24 May 2022
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity Yao Lu Max Bartolo Alastair Moore Sebastian Riedel Pontus Stenetorp AILaw LRM 279 1,120 0 18 Apr 2021