What Can Transformers Learn In-Context? A Case Study of Simple Function Classes

1 August 2022

Papers citing "What Can Transformers Learn In-Context? A Case Study of Simple Function Classes"

50 / 102 papers shown

Title
Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification Shang Liu Zhongze Cai Guanting Chen Xiaocheng Li UQCV 48 1 0 24 May 2024
How Do Transformers "Do" Physics? Investigating the Simple Harmonic Oscillator Subhash Kantamneni Ziming Liu Max Tegmark 19 2 0 23 May 2024
Where does In-context Translation Happen in Large Language Models Suzanna Sia David Mueller Kevin Duh LRM 41 0 0 07 Mar 2024
Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling Mahdi Karami Ali Ghodsi VLM 48 6 0 28 Feb 2024
Evaluating the Performance of ChatGPT for Spam Email Detection Shijing Si Yuwei Wu Jiawen Gu Yugui Zhang Jedrek Wosik Qinliang Su 59 8 0 23 Feb 2024
OmniPred: Language Models as Universal Regressors Xingyou Song Oscar Li Chansoo Lee Bangding Yang Daiyi Peng Sagi Perel Yutian Chen 60 14 0 22 Feb 2024
Linear Transformers are Versatile In-Context Learners Max Vladymyrov J. Oswald Mark Sandler Rong Ge 36 14 0 21 Feb 2024
An Information-Theoretic Analysis of In-Context Learning Hong Jun Jeon Jason D. Lee Qi Lei Benjamin Van Roy 29 19 0 28 Jan 2024
In-context Learning with Retrieved Demonstrations for Language Models: A Survey an Luo Xin Xu Yue Liu Panupong Pasupat Mehran Kazemi RALM 34 55 0 21 Jan 2024
FlexModel: A Framework for Interpretability of Distributed Large Language Models Matthew Choi Muhammad Adil Asif John Willes David Emerson AI4CE ALM 30 1 0 05 Dec 2023
Hot PATE: Private Aggregation of Distributions for Diverse Task Edith Cohen Benjamin Cohen-Wang Xin Lyu Jelani Nelson Tamas Sarlos Uri Stemmer 52 3 0 04 Dec 2023
The mechanistic basis of data dependence and abrupt learning in an in-context classification task Gautam Reddy 27 52 0 03 Dec 2023
Positional Information Matters for Invariant In-Context Learning: A Case Study of Simple Function Classes Yongqiang Chen Binghui Xie Kaiwen Zhou Bo Han Yatao Bian James Cheng 35 3 0 30 Nov 2023
Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks Rahul Ramesh Ekdeep Singh Lubana Mikail Khona Robert P. Dick Hidenori Tanaka CoGe 39 7 0 21 Nov 2023
Looped Transformers are Better at Learning Learning Algorithms Liu Yang Kangwook Lee Robert D. Nowak Dimitris Papailiopoulos 29 24 0 21 Nov 2023
Transformers are Provably Optimal In-context Estimators for Wireless Communications Vishnu Teja Kunde Vicram Rajagopalan Chandra Shekhara Kaushik Valmeekam Krishna R. Narayanan S. Shakkottai D. Kalathil J. Chamberland 37 4 0 01 Nov 2023
The Expressibility of Polynomial based Attention Scheme Zhao Song Guangyi Xu Junze Yin 34 5 0 30 Oct 2023
LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis Shih-Chieh Dai Aiping Xiong Lun-Wei Ku 27 64 0 23 Oct 2023
Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning Lucas Weber Elia Bruni Dieuwke Hupkes 32 25 0 20 Oct 2023
Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining Licong Lin Yu Bai Song Mei OffRL 32 45 0 12 Oct 2023
Fine-tune Language Models to Approximate Unbiased In-context Learning Timothy Chu Zhao Song Chiwun Yang 29 15 0 05 Oct 2023
GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Models Yonggan Fu Yongan Zhang Zhongzhi Yu Sixu Li Zhifan Ye Chaojian Li Cheng Wan Ying Lin 46 64 0 19 Sep 2023
Investigating the Learning Behaviour of In-context Learning: A Comparison with Supervised Learning Xindi Wang Yufei Wang Can Xu Xiubo Geng Bowen Zhang Chongyang Tao Frank Rudzicz Robert E. Mercer Daxin Jiang 33 11 0 28 Jul 2023
Advances and Challenges in Meta-Learning: A Technical Review Anna Vettoruzzo Mohamed-Rafik Bouguelia Joaquin Vanschoren Thorsteinn Rögnvaldsson K. Santosh OffRL 29 70 0 10 Jul 2023
Large Language Models as General Pattern Machines Suvir Mirchandani F. Xia Peter R. Florence Brian Ichter Danny Driess Montse Gonzalez Arenas Kanishka Rao Dorsa Sadigh Andy Zeng LLMAG 59 185 0 10 Jul 2023
Schema-learning and rebinding as mechanisms of in-context learning and emergence Siva K. Swaminathan Antoine Dedieu Rajkumar Vasudeva Raju Murray Shanahan Miguel Lazaro-Gredilla Dileep George 36 9 0 16 Jun 2023
Memorization Capacity of Multi-Head Attention in Transformers Sadegh Mahdavi Renjie Liao Christos Thrampoulidis 26 22 0 03 Jun 2023
Taming AI Bots: Controllability of Neural States in Large Language Models Stefano Soatto Paulo Tabuada Pratik Chaudhari Tianwei Liu LLMAG LM&Ro 18 13 0 29 May 2023
Im-Promptu: In-Context Composition from Image Prompts Bhishma Dedhia Michael Chang Jake C. Snell Thomas Griffiths N. Jha LRM MLLM 32 1 0 26 May 2023
A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Tasks Jacob D. Abernethy Alekh Agarwal T. V. Marinov Manfred K. Warmuth 28 18 0 26 May 2023
Active Learning Principles for In-Context Learning with Large Language Models Katerina Margatina Timo Schick Nikolaos Aletras Jane Dwivedi-Yu 32 39 0 23 May 2023
Brain-inspired learning in artificial neural networks: a review Samuel Schmidgall Jascha Achterberg Thomas Miconi Louis Kirsch Rojin Ziaei S. P. Hajiseyedrazi Jason Eshraghian 31 52 0 18 May 2023
Towards Robust Prompts on Vision-Language Models Jindong Gu Ahmad Beirami Xuezhi Wang Alex Beutel Philip Torr Yao Qin VLM VPVLM 38 8 0 17 Apr 2023
Larger language models do in-context learning differently Jerry W. Wei Jason W. Wei Yi Tay Dustin Tran Albert Webson ... Xinyun Chen Hanxiao Liu Da Huang Denny Zhou Tengyu Ma ReLM LRM 49 354 0 07 Mar 2023
Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following Seonghyeon Ye Hyeonbin Hwang Sohee Yang Hyeongu Yun Yireun Kim Minjoon Seo LRM 32 34 0 28 Feb 2023
Learning How to Infer Partial MDPs for In-Context Adaptation and Exploration Chentian Jiang Nan Rosemary Ke Hado van Hasselt 16 3 0 08 Feb 2023
Learning Functional Transduction Mathieu Chalvidal Thomas Serre Rufin VanRullen AI4CE 38 2 0 01 Feb 2023
An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models Yufeng Zhang Boyi Liu Qi Cai Lingxiao Wang Zhaoran Wang 53 11 0 30 Dec 2022
Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers Damai Dai Yutao Sun Li Dong Y. Hao Shuming Ma Zhifang Sui Furu Wei LRM 23 150 0 20 Dec 2022
General-Purpose In-Context Learning by Meta-Learning Transformers Louis Kirsch James Harrison Jascha Narain Sohl-Dickstein Luke Metz 40 72 0 08 Dec 2022
Teaching Structured Vision&Language Concepts to Vision&Language Models Sivan Doveh Assaf Arbelle Sivan Harary Yikang Shen Roei Herzig ... Donghyun Kim Raja Giryes Rogerio Feris S. Ullman Leonid Karlinsky VLM CoGe 56 70 0 21 Nov 2022
Prompting PaLM for Translation: Assessing Strategies and Performance David Vilar Markus Freitag Colin Cherry Jiaming Luo Viresh Ratnakar George F. Foster LRM 27 154 0 16 Nov 2022
Robustness of Demonstration-based Learning Under Limited Data Scenario Hongxin Zhang Yanzhe Zhang Ruiyi Zhang Diyi Yang 42 13 0 19 Oct 2022
Revision Transformers: Instructing Language Models to Change their Values Felix Friedrich Wolfgang Stammer P. Schramowski Kristian Kersting KELM 33 6 0 19 Oct 2022
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 250 463 0 24 Sep 2022
Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans John J. Nay ELM AILaw 88 27 0 14 Sep 2022
Recurrent Convolutional Neural Networks Learn Succinct Learning Algorithms Surbhi Goel Sham Kakade Adam Tauman Kalai Cyril Zhang 34 1 0 01 Sep 2022
Meta-learning via Language Model In-context Tuning Yanda Chen Ruiqi Zhong Sheng Zha George Karypis He He 236 157 0 15 Oct 2021
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity Yao Lu Max Bartolo Alastair Moore Sebastian Riedel Pontus Stenetorp AILaw LRM 279 1,125 0 18 Apr 2021
What Makes Good In-Context Examples for GPT- $3$ ? Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin Weizhu Chen AAML RALM 275 1,313 0 17 Jan 2021