What Can Transformers Learn In-Context? A Case Study of Simple Function Classes

1 August 2022

Papers citing "What Can Transformers Learn In-Context? A Case Study of Simple Function Classes"

50 / 91 papers shown

Title
Towards Contamination Resistant Benchmarks Rahmatullah Musawi Sheng Lu 42 0 0 13 May 2025
Understanding In-context Learning of Addition via Activation Subspaces Xinyan Hu Kayo Yin Michael I. Jordan Jacob Steinhardt Lijie Chen 53 0 0 08 May 2025
Rethinking Invariance in In-context Learning Lizhe Fang Yifei Wang Khashayar Gatmiry Lei Fang Yueming Wang 54 2 0 08 May 2025
Quiet Feature Learning in Algorithmic Tasks Prudhviraj Naidu Zixian Wang Leon Bergen R. Paturi VLM 57 0 0 06 May 2025
MateICL: Mitigating Attention Dispersion in Large-Scale In-Context Learning Murtadha Ahmed Wenbo Liu yunfeng 41 0 0 02 May 2025
On the generalization of language models from in-context learning and finetuning: a controlled study Andrew Kyle Lampinen Arslan Chaudhry Stephanie Chan Cody Wild Diane Wan Alex Ku Jorg Bornschein Razvan Pascanu Murray Shanahan James L. McClelland 46 0 0 01 May 2025
ICL CIPHERS: Quantifying "Learning'' in In-Context Learning via Substitution Ciphers Zhouxiang Fang Aayush Mishra Muhan Gao Anqi Liu Daniel Khashabi 44 0 0 28 Apr 2025
How Private is Your Attention? Bridging Privacy with In-Context Learning Soham Bonnerjee Zhen Wei Yeon Anna Asch Sagnik Nandy Promit Ghosal 49 0 0 22 Apr 2025
M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning in Large Vision-Language Models Yanshu Li Hongyang He Yi Cao Qisen Cheng Xiang Fu Ruixiang Tang VLM 40 0 0 06 Apr 2025
An extension of linear self-attention for in-context learning Katsuyuki Hagiwara 41 0 0 31 Mar 2025
Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations Yanshu Li 44 0 0 05 Mar 2025
Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning Anh Tong Thanh Nguyen-Tang Dongeun Lee Duc Nguyen Toan M. Tran David Hall Cheongwoong Kang Jaesik Choi 35 0 0 03 Mar 2025
In-Context Learning with Hypothesis-Class Guidance Ziqian Lin Shubham Kumar Bharti Kangwook Lee 76 0 0 27 Feb 2025
CoT-ICL Lab: A Petri Dish for Studying Chain-of-Thought Learning from In-Context Demonstrations Vignesh Kothapalli Hamed Firooz Maziar Sanjabi 68 0 0 21 Feb 2025
Zero-shot Model-based Reinforcement Learning using Large Language Models Abdelhakim Benechehab Youssef Attia El Hili Ambroise Odonnat Oussama Zekri Albert Thomas Giuseppe Paolo Maurizio Filippone I. Redko Balázs Kégl OffRL 72 1 0 17 Feb 2025
360Brew: A Decoder-only Foundation Model for Personalized Ranking and Recommendation Hamed Firooz Maziar Sanjabi Adrian Englhardt Aman Gupta Ben Levine ... Xiaoling Zhai Ya Xu Yu Wang Yun Dai Yun Dai ALM 47 3 0 27 Jan 2025
Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? Yutong Yin Zhaoran Wang LRM ReLM 146 0 0 27 Jan 2025
Out-of-distribution generalization via composition: a lens through induction heads in Transformers Jiajun Song Zhuoyan Xu Yiqiao Zhong 88 4 0 31 Dec 2024
ICLR: In-Context Learning of Representations Core Francisco Park Andrew Lee Ekdeep Singh Lubana Yongyi Yang Maya Okawa Kento Nishi Martin Wattenberg Hidenori Tanaka AIFin 120 3 0 29 Dec 2024
Toward Understanding In-context vs. In-weight Learning Bryan Chan Xinyi Chen András Gyorgy Dale Schuurmans 75 4 0 30 Oct 2024
In-context learning and Occam's razor Eric Elmoznino Tom Marty Tejas Kasetty Léo Gagnon Sarthak Mittal Mahan Fathi Dhanya Sridhar Guillaume Lajoie 42 1 0 17 Oct 2024
On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery Renpu Liu Ruida Zhou Cong Shen Jing Yang 30 0 0 17 Oct 2024
Context-Scaling versus Task-Scaling in In-Context Learning Amirhesam Abedsoltan Adityanarayanan Radhakrishnan Jingfeng Wu M. Belkin ReLM LRM 40 3 0 16 Oct 2024
State-space models can learn in-context by gradient descent Neeraj Mohan Sushma Yudou Tian Harshvardhan Mestha Nicolo Colombo David Kappel Anand Subramoney 41 3 0 15 Oct 2024
Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent Bo Chen Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song 96 20 0 15 Oct 2024
Zero-Shot Learning of Causal Models Divyat Mahajan Jannes Gladrow Agrin Hilmkil Cheng Zhang M. Scetbon 42 1 0 08 Oct 2024
In-context Learning in Presence of Spurious Correlations Hrayr Harutyunyan R. Darbinyan Samvel Karapetyan Hrant Khachatrian LRM 51 1 0 04 Oct 2024
Towards Understanding the Universality of Transformers for Next-Token Prediction Michael E. Sander Gabriel Peyré CML 39 0 0 03 Oct 2024
On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding Kevin Xu Issei Sato 39 3 0 02 Oct 2024
Transformers Handle Endogeneity in In-Context Linear Regression Haodong Liang Krishnakumar Balasubramanian Lifeng Lai 41 1 0 02 Oct 2024
State space models, emergence, and ergodicity: How many parameters are needed for stable predictions? Ingvar M. Ziemann Nikolai Matni George J. Pappas 27 1 0 20 Sep 2024
MILE: A Mutation Testing Framework of In-Context Learning Systems Zeming Wei Yihao Zhang Meng Sun 48 0 0 07 Sep 2024
Spin glass model of in-context learning Yuhao Li Ruoran Bai Haiping Huang LRM 47 0 0 05 Aug 2024
Representing Rule-based Chatbots with Transformers Dan Friedman Abhishek Panigrahi Danqi Chen 66 1 0 15 Jul 2024
From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning Nan Xu Fei Wang Sheng Zhang Hoifung Poon Muhao Chen 34 6 0 01 Jul 2024
Expressivity of Neural Networks with Random Weights and Learned Biases Ezekiel Williams Avery Hee-Woon Ryoo Thomas Jiralerspong Alexandre Payeur M. Perich Luca Mazzucato Guillaume Lajoie 36 2 0 01 Jul 2024
Separations in the Representational Capabilities of Transformers and Recurrent Architectures S. Bhattamishra Michael Hahn Phil Blunsom Varun Kanade GNN 44 9 0 13 Jun 2024
From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems Jianliang He Siyu Chen Fengzhuo Zhang Zhuoran Yang LM&Ro LLMAG 44 2 0 30 May 2024
Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting Suraj Anand Michael A. Lepori Jack Merullo Ellie Pavlick CLL 33 6 0 28 May 2024
Benchmarking General-Purpose In-Context Learning Fan Wang Chuan Lin Yang Cao Yu Kang 38 1 0 27 May 2024
On Understanding Attention-Based In-Context Learning for Categorical Data Aaron T. Wang William Convertino Xiang Cheng Ricardo Henao Lawrence Carin 61 0 0 27 May 2024
Unsupervised Meta-Learning via In-Context Learning Anna Vettoruzzo Lorenzo Braccaioli Joaquin Vanschoren M. Nowaczyk SSL 64 0 0 25 May 2024
Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification Shang Liu Zhongze Cai Guanting Chen Xiaocheng Li UQCV 46 1 0 24 May 2024
Where does In-context Translation Happen in Large Language Models Suzanna Sia David Mueller Kevin Duh LRM 41 0 0 07 Mar 2024
Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling Mahdi Karami Ali Ghodsi VLM 48 6 0 28 Feb 2024
Evaluating the Performance of ChatGPT for Spam Email Detection Shijing Si Yuwei Wu Jiawen Gu Yugui Zhang Jedrek Wosik Qinliang Su 49 8 0 23 Feb 2024
OmniPred: Language Models as Universal Regressors Xingyou Song Oscar Li Chansoo Lee Bangding Yang Daiyi Peng Sagi Perel Yutian Chen 56 14 0 22 Feb 2024
Linear Transformers are Versatile In-Context Learners Max Vladymyrov J. Oswald Mark Sandler Rong Ge 34 13 0 21 Feb 2024
An Information-Theoretic Analysis of In-Context Learning Hong Jun Jeon Jason D. Lee Qi Lei Benjamin Van Roy 27 18 0 28 Jan 2024
FlexModel: A Framework for Interpretability of Distributed Large Language Models Matthew Choi Muhammad Adil Asif John Willes David Emerson AI4CE ALM 27 1 0 05 Dec 2023