Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions

4 October 2023

Papers citing "Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions"

34 / 34 papers shown

Title
LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasures Francisco Aguilera-Martínez Fernando Berzal PILM 55 0 0 02 May 2025
From predictions to confidence intervals: an empirical study of conformal prediction methods for in-context learning Zhe Huang Simone Rossi Rui Yuan T. Hannagan 34 0 0 22 Apr 2025
In-Context Learning with Hypothesis-Class Guidance Ziqian Lin Shubham Kumar Bharti Kangwook Lee 76 0 0 27 Feb 2025
On the Robustness of Transformers against Context Hijacking for Linear Classification Tianle Li Chenyang Zhang Xingwu Chen Yuan Cao Difan Zou 72 0 0 24 Feb 2025
Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoning Xuetao Ma Wenbin Jiang Hua Huang LRM 68 1 0 24 Feb 2025
Vector-ICL: In-context Learning with Continuous Vector Representations Yufan Zhuang Chandan Singh Liyuan Liu Jingbo Shang Jianfeng Gao 54 3 0 21 Feb 2025
Solving Empirical Bayes via Transformers Anzo Teh Mark Jabbour Yury Polyanskiy 93 0 0 17 Feb 2025
Re-examining learning linear functions in context Omar Naim Guilhem Fouilhé Nicholas Asher 63 0 0 18 Nov 2024
Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning Qingyu Yin Xuzheng He Luoao Deng Chak Tou Leong Fan Wang Yanzhao Yan Xiaoyu Shen Qiang Zhang 39 2 0 07 Oct 2024
Task Diversity Shortens the ICL Plateau Jaeyeon Kim Sehyun Kwon Joo Young Choi Jongho Park Jaewoong Cho Jason D. Lee Ernest K. Ryu MoMe 36 2 0 07 Oct 2024
Cognitive LLMs: Towards Integrating Cognitive Architectures and Large Language Models for Manufacturing Decision-making Siyu Wu A. Oltramari Jonathan M Francis C. L. Giles Frank E. Ritter 48 0 0 17 Aug 2024
Sub-SA: Strengthen In-context Learning via Submodular Selective Annotation Jian Qian Miao Sun Sifan Zhou Ziyu Zhao Ruizhi Hun Patrick Chiang 41 1 0 08 Jul 2024
Separations in the Representational Capabilities of Transformers and Recurrent Architectures S. Bhattamishra Michael Hahn Phil Blunsom Varun Kanade GNN 44 9 0 13 Jun 2024
XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning Alexander Nikulin Ilya Zisman Alexey Zemtsov Viacheslav Sinii 113 4 0 13 Jun 2024
On Limitation of Transformer for Learning HMMs Jiachen Hu Qinghua Liu Chi Jin 50 3 0 06 Jun 2024
A Theoretical Understanding of Self-Correction through In-context Alignment Yifei Wang Yuyang Wu Zeming Wei Stefanie Jegelka Yisen Wang LRM 47 13 0 28 May 2024
On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability Chenyu Zheng Wei Huang Rongzheng Wang Guoqiang Wu Jun Zhu Chongxuan Li 39 1 0 27 May 2024
DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning Zijian Zhou Xiaoqiang Lin Xinyi Xu Alok Prakash Daniela Rus K. H. Low 36 2 0 22 May 2024
State-Free Inference of State-Space Models: The Transfer Function Approach Rom N. Parnichkun Stefano Massaroli Alessandro Moro Jimmy T.H. Smith Ramin Hasani ... Hajime Asama Stefano Ermon Taiji Suzuki Atsushi Yamashita Michael Poli 44 5 0 10 May 2024
Exploring the Robustness of In-Context Learning with Noisy Labels Chen Cheng Xinzhi Yu Haodong Wen Jinsong Sun Guanzhang Yue Yihao Zhang Zeming Wei NoLa 24 6 0 28 Apr 2024
Many-Shot In-Context Learning Rishabh Agarwal Avi Singh Lei M. Zhang Bernd Bohnet Luis Rosias ... John D. Co-Reyes Eric Chu Feryal M. P. Behbahani Aleksandra Faust Hugo Larochelle ReLM OffRL BDL 63 96 0 17 Apr 2024
TEL'M: Test and Evaluation of Language Models G. Cybenko Joshua Ackerman Paul Lintilhac ALM ELM 40 0 0 16 Apr 2024
From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples Robert Vacareanu Vlad-Andrei Negru Vasile Suciu Mihai Surdeanu 31 28 0 11 Apr 2024
Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models Sebastian Bordt Harsha Nori Vanessa Rodrigues Besmira Nushi Rich Caruana 38 13 0 09 Apr 2024
Mechanistic Design and Scaling of Hybrid Architectures Michael Poli Armin W. Thomas Eric N. D. Nguyen Pragaash Ponnusamy Bjorn Deiseroth ... Brian Hie Stefano Ermon Christopher Ré Ce Zhang Stefano Massaroli MoE 57 21 0 26 Mar 2024
Can large language models explore in-context? Akshay Krishnamurthy Keegan Harris Dylan J. Foster Cyril Zhang Aleksandrs Slivkins LM&Ro LLMAG LRM 126 23 0 22 Mar 2024
In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization Ruiqi Zhang Jingfeng Wu Peter L. Bartlett 38 12 0 22 Feb 2024
Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks Jongho Park Jaeseung Park Zheyang Xiong Nayoung Lee Jaewoong Cho Samet Oymak Kangwook Lee Dimitris Papailiopoulos 30 69 0 06 Feb 2024
In-Context Language Learning: Architectures and Algorithms Ekin Akyürek Bailin Wang Yoon Kim Jacob Andreas LRM ReLM 45 42 0 23 Jan 2024
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 250 463 0 24 Sep 2022
Neural Networks and the Chomsky Hierarchy Grégoire Delétang Anian Ruoss Jordi Grau-Moya Tim Genewein L. Wenliang ... Chris Cundy Marcus Hutter Shane Legg Joel Veness Pedro A. Ortega UQCV 107 131 0 05 Jul 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 339 12,003 0 04 Mar 2022
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity Yao Lu Max Bartolo Alastair Moore Sebastian Riedel Pontus Stenetorp AILaw LRM 279 1,124 0 18 Apr 2021
What Makes Good In-Context Examples for GPT- $3$ ? Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin Weizhu Chen AAML RALM 275 1,312 0 17 Jan 2021