A Theory of Emergent In-Context Learning as Implicit Structure Induction

14 March 2023

Papers citing "A Theory of Emergent In-Context Learning as Implicit Structure Induction"

50 / 56 papers shown

Title
ICL CIPHERS: Quantifying "Learning'' in In-Context Learning via Substitution Ciphers Zhouxiang Fang Aayush Mishra Muhan Gao Anqi Liu Daniel Khashabi 44 0 0 28 Apr 2025
When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars Rei Higuchi Ryotaro Kawata Naoki Nishikawa Kazusato Oko Shoichiro Yamaguchi Sosuke Kobayashi Seiya Tokui K. Hayashi Daisuke Okanohara Taiji Suzuki AI4CE 40 0 0 24 Apr 2025
Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B Aleksandra Bakalova Yana Veitsman Xinting Huang Michael Hahn 33 0 0 31 Mar 2025
Enough Coin Flips Can Make LLMs Act Bayesian Ritwik Gupta Rodolfo Corona Jiaxin Ge Eric Wang Dan Klein Trevor Darrell David M. Chan LRM BDL 50 1 0 06 Mar 2025
Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers Alireza Amiri Xinting Huang Mark Rofin Michael Hahn LRM 192 0 0 04 Feb 2025
Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? Yutong Yin Zhaoran Wang LRM ReLM 158 0 0 27 Jan 2025
Using Pre-trained LLMs for Multivariate Time Series Forecasting Malcolm Wolff Shenghao Yang Kari Torkkola Michael W. Mahoney AI4TS AIFin 46 1 0 10 Jan 2025
Out-of-distribution generalization via composition: a lens through induction heads in Transformers Jiajun Song Zhuoyan Xu Yiqiao Zhong 88 4 0 31 Dec 2024
Conceptual In-Context Learning and Chain of Concepts: Solving Complex Conceptual Problems Using Large Language Models Nishtha N. Vaidya Thomas Runkler Thomas Hubauer Veronika Haderlein-Hoegberg Maja Mlicic Brandt LRM 80 0 0 19 Dec 2024
Bayesian scaling laws for in-context learning Aryaman Arora Dan Jurafsky Christopher Potts Noah D. Goodman 29 2 0 21 Oct 2024
On the Training Convergence of Transformers for In-Context Classification Wei Shen Ruida Zhou Jing Yang Cong Shen 28 3 0 15 Oct 2024
Racing Thoughts: Explaining Contextualization Errors in Large Language Models Michael A. Lepori Michael Mozer Asma Ghandeharioun LRM 85 1 0 02 Oct 2024
In-Context Learning with Representations: Contextual Generalization of Trained Transformers Tong Yang Yu Huang Yingbin Liang Yuejie Chi MLT 40 5 0 19 Aug 2024
Representing Rule-based Chatbots with Transformers Dan Friedman Abhishek Panigrahi Danqi Chen 66 1 0 15 Jul 2024
Estimating the Hallucination Rate of Generative AI Andrew Jesson Nicolas Beltran-Velez Quentin Chu Sweta Karlekar Jannik Kossen Yarin Gal John P. Cunningham David M. Blei 51 6 0 11 Jun 2024
On Subjective Uncertainty Quantification and Calibration in Natural Language Generation Ziyu Wang Chris Holmes UQLM 53 4 0 07 Jun 2024
What Do Language Models Learn in Context? The Structured Task Hypothesis Jiaoda Li Yifan Hou Mrinmaya Sachan Ryan Cotterell LRM 44 7 0 06 Jun 2024
Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective Fabian Falck Ziyu Wang Chris Holmes 58 13 0 02 Jun 2024
From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems Jianliang He Siyu Chen Fengzhuo Zhang Zhuoran Yang LM&Ro LLMAG 44 2 0 30 May 2024
Does learning the right latent variables necessarily improve in-context learning? Sarthak Mittal Eric Elmoznino Léo Gagnon Sangnie Bhardwaj Dhanya Sridhar Guillaume Lajoie 34 4 0 29 May 2024
Finding Visual Task Vectors Alberto Hojel Yutong Bai Trevor Darrell Amir Globerson Amir Bar 70 6 0 08 Apr 2024
Can large language models explore in-context? Akshay Krishnamurthy Keegan Harris Dylan J. Foster Cyril Zhang Aleksandrs Slivkins LM&Ro LLMAG LRM 126 23 0 22 Mar 2024
Concept-aware Data Construction Improves In-context Learning of Language Models Michal Štefánik Marek Kadlcík Petr Sojka 54 0 0 08 Mar 2024
LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational History Akash Gupta Ivaxi Sheth Vyas Raina Mark J. F. Gales Mario Fritz 43 4 0 28 Feb 2024
Visual In-Context Learning for Large Vision-Language Models Yucheng Zhou Xiang Li Qianning Wang Jianbing Shen MLLM 27 58 0 18 Feb 2024
Understanding In-Context Learning with a Pelican Soup Framework Ting-Rui Chiang Dani Yogatama 16 1 0 16 Feb 2024
Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks Jongho Park Jaeseung Park Zheyang Xiong Nayoung Lee Jaewoong Cho Samet Oymak Kangwook Lee Dimitris Papailiopoulos 30 69 0 06 Feb 2024
Learning Universal Predictors Jordi Grau-Moya Tim Genewein Marcus Hutter Laurent Orseau Grégoire Delétang ... Anian Ruoss Wenliang Kevin Li Christopher Mattern Matthew Aitchison J. Veness 27 11 0 26 Jan 2024
Demystifying Chains, Trees, and Graphs of Thoughts Maciej Besta Florim Memedi Zhenyu Zhang Robert Gerstenberger Guangyuan Piao ... Aleš Kubíček H. Niewiadomski Aidan O'Mahony Onur Mutlu Torsten Hoefler AI4CE LRM 75 27 0 25 Jan 2024
In-Context Language Learning: Architectures and Algorithms Ekin Akyürek Bailin Wang Yoon Kim Jacob Andreas LRM ReLM 45 42 0 23 Jan 2024
Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning Shuai Zhao Meihuizi Jia Anh Tuan Luu Fengjun Pan Jinming Wen AAML 31 36 0 11 Jan 2024
Generalization to New Sequential Decision Making Tasks with In-Context Learning Sharath Chandra Raparthy Eric Hambro Robert Kirk Mikael Henaff Roberta Raileanu OffRL 111 21 0 06 Dec 2023
How are Prompts Different in Terms of Sensitivity? Sheng Lu Hendrik Schuff Iryna Gurevych 40 18 0 13 Nov 2023
Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptions Sachin Kumar Chan Young Park Yulia Tsvetkov VLM 30 2 0 13 Nov 2023
The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis Yuxiang Zhou Jiazheng Li Yanzheng Xiang Hanqi Yan Lin Gui Yulan He 24 14 0 01 Nov 2023
Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selection Costas Mavromatis Balasubramaniam Srinivasan Zhengyuan Shen Jiani Zhang Huzefa Rangwala Christos Faloutsos George Karypis 19 21 0 30 Oct 2023
In-Context Learning Dynamics with Random Binary Sequences Eric J. Bigelow Ekdeep Singh Lubana Robert P. Dick Hidenori Tanaka T. Ullman 34 4 0 26 Oct 2023
Function Vectors in Large Language Models Eric Todd Millicent Li Arnab Sen Sharma Aaron Mueller Byron C. Wallace David Bau 14 100 0 23 Oct 2023
Do pretrained Transformers Learn In-Context by Gradient Descent? Lingfeng Shen Aayush Mishra Daniel Khashabi 39 7 0 12 Oct 2023
Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions S. Bhattamishra Arkil Patel Phil Blunsom Varun Kanade 24 41 0 04 Oct 2023
Understanding In-Context Learning from Repetitions Jianhao Yan Jin Xu Chiyu Song Chenming Wu Yafu Li Yue Zhang 30 20 0 30 Sep 2023
Are Emergent Abilities in Large Language Models just In-Context Learning? Sheng Lu Irina Bigoulaeva Rachneet Sachdeva Harish Tayyar Madabushi Iryna Gurevych LRM ELM ReLM 51 93 0 04 Sep 2023
Uncertainty in Natural Language Generation: From Theory to Applications Joris Baan Nico Daheim Evgenia Ilia Dennis Ulmer Haau-Sing Li Raquel Fernández Barbara Plank Rico Sennrich Chrysoula Zerva Wilker Aziz UQLM 34 40 0 28 Jul 2023
In-Context Learning Learns Label Relationships but Is Not Conventional Learning Jannik Kossen Y. Gal Tom Rainforth 37 27 0 23 Jul 2023
Large Language Models Michael R Douglas LLMAG LM&MA 42 558 0 11 Jul 2023
Trainable Transformer in Transformer A. Panigrahi Sadhika Malladi Mengzhou Xia Sanjeev Arora VLM 32 12 0 03 Jul 2023
In-Context Learning through the Bayesian Prism Madhuri Panwar Kabir Ahuja Navin Goyal BDL 39 39 0 08 Jun 2023
What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization Yufeng Zhang Fengzhuo Zhang Zhuoran Yang Zhaoran Wang BDL 36 63 0 30 May 2023
Dissecting Chain-of-Thought: Compositionality through In-Context Filtering and Learning Yingcong Li Kartik K. Sreenivasan Angeliki Giannou Dimitris Papailiopoulos Samet Oymak LRM 16 16 0 30 May 2023
Concept-aware Training Improves In-context Learning Ability of Language Models Michal Štefánik Marek Kadlcík KELM LRM 38 0 0 23 May 2023