An Information-Theoretic Analysis of In-Context Learning

An Information-Theoretic Analysis of In-Context Learning

28 January 2024

Benjamin Van Roy

Papers citing "An Information-Theoretic Analysis of In-Context Learning"

18 / 18 papers shown

Title
HyperTree Planning: Enhancing LLM Reasoning via Hierarchical Thinking Runquan Gui Z. Wang J. Wang Chi Ma Huiling Zhen M. Yuan Jianye Hao Defu Lian Enhong Chen Feng Wu LRM 133 0 0 05 May 2025
Gating is Weighting: Understanding Gated Linear Attention through In-context Learning Yingcong Li Davoud Ataee Tarzanagh A. S. Rawat Maryam Fazel Samet Oymak 25 0 0 06 Apr 2025
Pretrained transformer efficiently learns low-dimensional target functions in-context Kazusato Oko Yujin Song Taiji Suzuki Denny Wu 39 4 0 04 Nov 2024
A Theoretical Survey on Foundation Models Shi Fu Yuzhu Chen Yingjie Wang Dacheng Tao 26 0 0 15 Oct 2024
Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Information Yongheng Zhang Qiguang Chen Jingxuan Zhou Peng Wang Jiasheng Si Jin Wang Wenpeng Lu Libo Qin LRM 46 3 0 06 Oct 2024
Large Language Models as Markov Chains Oussama Zekri Ambroise Odonnat Abdelhakim Benechehab Linus Bleistein Nicolas Boullé I. Redko 40 9 0 03 Oct 2024
In-Context Learning with Representations: Contextual Generalization of Trained Transformers Tong Yang Yu Huang Yingbin Liang Yuejie Chi MLT 32 5 0 19 Aug 2024
Estimating the Hallucination Rate of Generative AI Andrew Jesson Nicolas Beltran-Velez Quentin Chu Sweta Karlekar Jannik Kossen Yarin Gal John P. Cunningham David M. Blei 46 6 0 11 Jun 2024
Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification Shang Liu Zhongze Cai Guanting Chen Xiaocheng Li UQCV 40 1 0 24 May 2024
Understanding the Training and Generalization of Pretrained Transformer for Sequential Decision Making Hanzhao Wang Yu Pan Fupeng Sun Shang Liu K. Talluri Guanting Chen Xiaocheng Li OffRL 53 1 0 23 May 2024
Many-Shot In-Context Learning Rishabh Agarwal Avi Singh Lei M. Zhang Bernd Bohnet Luis Rosias ... John D. Co-Reyes Eric Chu Feryal M. P. Behbahani Aleksandra Faust Hugo Larochelle ReLM OffRL BDL 55 96 0 17 Apr 2024
Can large language models explore in-context? Akshay Krishnamurthy Keegan Harris Dylan J. Foster Cyril Zhang Aleksandrs Slivkins LM&Ro LLMAG LRM 118 22 0 22 Mar 2024
Mechanics of Next Token Prediction with Self-Attention Yingcong Li Yixiao Huang M. E. Ildiz A. S. Rawat Samet Oymak 32 25 0 12 Mar 2024
Emergence of In-Context Reinforcement Learning from Noise Distillation Ilya Zisman Vladislav Kurenkov Alexander Nikulin Viacheslav Sinii Sergey Kolesnikov OffRL 33 9 0 19 Dec 2023
The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis Yuxiang Zhou Jiazheng Li Yanzheng Xiang Hanqi Yan Lin Gui Yulan He 22 14 0 01 Nov 2023
How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding Yuchen Li Yuan-Fang Li Andrej Risteski 117 61 0 07 Mar 2023
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity Yao Lu Max Bartolo Alastair Moore Sebastian Riedel Pontus Stenetorp AILaw LRM 277 1,120 0 18 Apr 2021
What Makes Good In-Context Examples for GPT- $3$ ? Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin Weizhu Chen AAML RALM 275 1,312 0 17 Jan 2021