General-Purpose In-Context Learning by Meta-Learning Transformers

8 December 2022

Louis Kirsch

James Harrison

Jascha Narain Sohl-Dickstein

Luke Metz

ArXiv PDF HTML

Papers citing "General-Purpose In-Context Learning by Meta-Learning Transformers"

50 / 66 papers shown

Title
How Private is Your Attention? Bridging Privacy with In-Context Learning Soham Bonnerjee Zhen Wei Yeon Anna Asch Sagnik Nandy Promit Ghosal 47 0 0 22 Apr 2025
Free Random Projection for In-Context Reinforcement Learning Tomohiro Hayase B. Collins Nakamasa Inoue 26 0 0 09 Apr 2025
Measuring In-Context Computation Complexity via Hidden State Prediction Vincent Herrmann Róbert Csordás Jürgen Schmidhuber 41 0 0 17 Mar 2025
Can We Optimize Deep RL Policy Weights as Trajectory Modeling? Hongyao Tang OffRL 77 0 0 06 Mar 2025
AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers Jake Grigsby Justin Sasek Samyak Parajuli Daniel Adebi Amy Zhang Yuke Zhu OffRL 26 3 0 17 Nov 2024
On the Role of Depth and Looping for In-Context Learning with Task Diversity Khashayar Gatmiry Nikunj Saunshi Sashank J. Reddi Stefanie Jegelka Sanjiv Kumar 29 2 0 29 Oct 2024
In-context learning and Occam's razor Eric Elmoznino Tom Marty Tejas Kasetty Léo Gagnon Sarthak Mittal Mahan Fathi Dhanya Sridhar Guillaume Lajoie 39 1 0 17 Oct 2024
Retrieval-Augmented Decision Transformer: External Memory for In-context RL Thomas Schmied Fabian Paischer Vihang Patil M. Hofmarcher Razvan Pascanu Sepp Hochreiter OffRL 44 6 0 09 Oct 2024
ConML: A Universal Meta-Learning Framework with Task-Level Contrastive Learning Shiguang Wu Yaqing Wang Yatao Bian Quanming Yao CLL 28 0 0 08 Oct 2024
Transformers learn variable-order Markov chains in-context Ruida Zhou C. Tian Suhas Diggavi 26 0 0 07 Oct 2024
Task Diversity Shortens the ICL Plateau Jaeyeon Kim Sehyun Kwon Joo Young Choi Jongho Park Jaewoong Cho Jason D. Lee Ernest K. Ryu MoMe 31 2 0 07 Oct 2024
In-context Learning in Presence of Spurious Correlations Hrayr Harutyunyan R. Darbinyan Samvel Karapetyan Hrant Khachatrian LRM 48 1 0 04 Oct 2024
Provable In-Context Learning of Linear Systems and Linear Elliptic PDEs with Transformers Frank Cole Yulong Lu Riley OÑeill Tianhao Zhang 45 2 0 18 Sep 2024
RoboMorph: In-Context Meta-Learning for Robot Dynamics Modeling Manuel Bianchi Bazzi Asad Ali Shahid Christopher Agia J. I. Alora Marco Forgione Dario Piga Francesco Braghin Marco Pavone L. Roveda LM&Ro AI4CE 30 0 0 18 Sep 2024
XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning Alexander Nikulin Ilya Zisman Alexey Zemtsov Viacheslav Sinii 110 4 0 13 Jun 2024
Benchmarking General-Purpose In-Context Learning Fan Wang Chuan Lin Yang Cao Yu Kang 32 1 0 27 May 2024
LoGAH: Predicting 774-Million-Parameter Transformers using Graph HyperNetworks with 1/100 Parameters Xinyu Zhou Boris Knyazev Alexia Jolicoeur-Martineau Jie Fu AI4CE 45 0 0 25 May 2024
Unsupervised Meta-Learning via In-Context Learning Anna Vettoruzzo Lorenzo Braccaioli Joaquin Vanschoren M. Nowaczyk SSL 61 0 0 25 May 2024
MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning Sanchit Sinha Yuguang Yue Victor Soto Mayank Kulkarni Jianhua Lu Aidong Zhang LRM 46 4 0 19 May 2024
Structurally Flexible Neural Networks: Evolving the Building Blocks for General Agents J. Pedersen Erwan Plantec Eleni Nisioti Milton L. Montero Sebastian Risi 52 1 0 06 Apr 2024
Can large language models explore in-context? Akshay Krishnamurthy Keegan Harris Dylan J. Foster Cyril Zhang Aleksandrs Slivkins LM&Ro LLMAG LRM 120 23 0 22 Mar 2024
Towards Understanding the Relationship between In-context Learning and Compositional Generalization Sungjun Han Sebastian Padó CoGe 21 2 0 18 Mar 2024
SplAgger: Split Aggregation for Meta-Reinforcement Learning Jacob Beck Matthew Jackson Risto Vuorio Zheng Xiong Shimon Whiteson OffRL 26 2 0 05 Mar 2024
Language Agents as Optimizable Graphs Mingchen Zhuge Wenyi Wang Louis Kirsch Francesco Faccio Dmitrii Khizbullin Jürgen Schmidhuber LLMAG 29 19 0 26 Feb 2024
The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains Benjamin L. Edelman Ezra Edelman Surbhi Goel Eran Malach Nikolaos Tsilivis BDL 26 42 0 16 Feb 2024
An Information-Theoretic Analysis of In-Context Learning Hong Jun Jeon Jason D. Lee Qi Lei Benjamin Van Roy 27 18 0 28 Jan 2024
Emergence of In-Context Reinforcement Learning from Noise Distillation Ilya Zisman Vladislav Kurenkov Alexander Nikulin Viacheslav Sinii Sergey Kolesnikov OffRL 33 9 0 19 Dec 2023
On the adaptation of in-context learners for system identification Dario Piga F. Pura Marco Forgione 18 3 0 07 Dec 2023
Generalization to New Sequential Decision Making Tasks with In-Context Learning Sharath Chandra Raparthy Eric Hambro Robert Kirk Mikael Henaff Roberta Raileanu OffRL 108 21 0 06 Dec 2023
The mechanistic basis of data dependence and abrupt learning in an in-context classification task Gautam Reddy 24 50 0 03 Dec 2023
The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis Yuxiang Zhou Jiazheng Li Yanzheng Xiang Hanqi Yan Lin Gui Yulan He 24 14 0 01 Nov 2023
Are LSTMs Good Few-Shot Learners? Mike Huisman Thomas M. Moerland Aske Plaat Jan N. van Rijn VLM 18 7 0 22 Oct 2023
Context-Aware Meta-Learning Christopher Fifty Dennis Duan Ronald G. Junkins Ehsan Amid Jurij Leskovec Christopher Ré Sebastian Thrun LRM VLM MLLM 25 10 0 17 Oct 2023
How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations Tianyu Guo Wei Hu Song Mei Huan Wang Caiming Xiong Silvio Savarese Yu Bai 27 47 0 16 Oct 2023
Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency Zhihan Liu Hao Hu Shenao Zhang Hongyi Guo Shuqi Ke Boyi Liu Zhaoran Wang LLMAG LRM 33 33 0 29 Sep 2023
The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute Aleksandar Stanić Dylan R. Ashley Oleg Serikov Louis Kirsch Francesco Faccio Jürgen Schmidhuber Thomas Hofmann Imanol Schlag MoE 43 9 0 20 Sep 2023
Understanding Catastrophic Forgetting in Language Models via Implicit Inference Suhas Kotha Jacob Mitchell Springer Aditi Raghunathan CLL 42 57 0 18 Sep 2023
Breaking through the learning plateaus of in-context learning in Transformer Jingwen Fu Tao Yang Yuwang Wang Yan Lu Nanning Zheng 30 1 0 12 Sep 2023
Uncovering mesa-optimization algorithms in Transformers J. Oswald Eyvind Niklasson Maximilian Schlegel Seijin Kobayashi Nicolas Zucchet ... Mark Sandler Blaise Agüera y Arcas Max Vladymyrov Razvan Pascanu João Sacramento 24 53 0 11 Sep 2023
From system models to class models: An in-context learning paradigm Marco Forgione F. Pura Dario Piga 32 12 0 25 Aug 2023
In-Context Learning Learns Label Relationships but Is Not Conventional Learning Jannik Kossen Y. Gal Tom Rainforth 37 27 0 23 Jul 2023
Advances and Challenges in Meta-Learning: A Technical Review Anna Vettoruzzo Mohamed-Rafik Bouguelia Joaquin Vanschoren Thorsteinn Rögnvaldsson K. Santosh OffRL 24 70 0 10 Jul 2023
Large Language Models as General Pattern Machines Suvir Mirchandani F. Xia Peter R. Florence Brian Ichter Danny Driess Montse Gonzalez Arenas Kanishka Rao Dorsa Sadigh Andy Zeng LLMAG 54 184 0 10 Jul 2023
Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression Allan Raventós Mansheej Paul F. Chen Surya Ganguli 27 70 0 26 Jun 2023
Supervised Pretraining Can Learn In-Context Reinforcement Learning Jonathan Lee Annie Xie Aldo Pacchiano Yash Chandak Chelsea Finn Ofir Nachum Emma Brunskill OffRL 35 74 0 26 Jun 2023
AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks Alexander Tornede Difan Deng Theresa Eimer Joseph Giovanelli Aditya Mohan ... Sarah Segel Daphne Theodorakopoulos Tanja Tornede Henning Wachsmuth Marius Lindauer 28 23 0 13 Jun 2023
A Closer Look at In-Context Learning under Distribution Shifts Kartik Ahuja David Lopez-Paz 37 14 0 26 May 2023
Brain-inspired learning in artificial neural networks: a review Samuel Schmidgall Jascha Achterberg Thomas Miconi Louis Kirsch Rojin Ziaei S. P. Hajiseyedrazi Jason Eshraghian 31 52 0 18 May 2023
Statistical Foundations of Prior-Data Fitted Networks T. Nagler AI4CE 24 20 0 18 May 2023
What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning Jane Pan Tianyu Gao Howard Chen Danqi Chen 28 108 0 16 May 2023