ResearchTrend.AI
In-context Learning and Induction Heads

In-context Learning and Induction Heads

Catherine Olsson
Nelson Elhage
Neel Nanda
Nicholas Joseph
Nova Dassarma
T. Henighan
Benjamin Mann
Amanda Askell
Yuntao Bai
Anna Chen
Tom Conerly
Dawn Drain
Deep Ganguli
Zac Hatfield-Dodds
Danny Hernandez
Scott R. Johnston
Andy Jones
John Kernion
Liane Lovitt
Kamal Ndousse
Dario Amodei
Tom B. Brown
Jack Clark
Jared Kaplan
Sam McCandlish
C. Olah
ArXiv (abs)PDFHTML

Papers citing "In-context Learning and Induction Heads"

50 / 434 papers shown
Title
From Concepts to Components: Concept-Agnostic Attention Module Discovery in Transformers
From Concepts to Components: Concept-Agnostic Attention Module Discovery in Transformers
Jingtong Su
Julia Kempe
Karen Ullrich
14
0
0
20 Jun 2025
Latent Concept Disentanglement in Transformer-based Language Models
Latent Concept Disentanglement in Transformer-based Language Models
Guan Zhe Hong
Bhavya Vasudeva
Vatsal Sharan
Cyrus Rashtchian
Prabhakar Raghavan
Rina Panigrahy
ReLMLRM
15
0
0
20 Jun 2025
Hidden Breakthroughs in Language Model Training
Hidden Breakthroughs in Language Model Training
Sara Kangaslahti
Elan Rosenfeld
Naomi Saphra
26
0
0
18 Jun 2025
Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View
Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View
Muhammad Cendekia Airlangga
Hilal AlQuabeh
Munachiso Nwadike
Kentaro Inui
KELM
12
0
0
18 Jun 2025
Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning
Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning
Chengye Li
Haiyun Liu
Yuanxi Li
15
0
0
13 Jun 2025
Interpreting learned search: finding a transition model and value function in an RNN that plays Sokoban
Interpreting learned search: finding a transition model and value function in an RNN that plays Sokoban
Mohammad Taufeeque
Aaron David Tucker
Adam Gleave
Adrià Garriga-Alonso
35
0
0
11 Jun 2025
Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking
Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking
Wuwei Zhang
Fangcong Yin
Howard Yen
Danqi Chen
Xi Ye
LRM
79
0
0
11 Jun 2025
Did I Faithfully Say What I Thought? Bridging the Gap Between Neural Activity and Self-Explanations in Large Language Models
Did I Faithfully Say What I Thought? Bridging the Gap Between Neural Activity and Self-Explanations in Large Language Models
Milan Bhan
Jean-Noel Vittaut
Nicolas Chesneau
Sarath Chandar
Marie-Jeanne Lesot
LRM
31
0
0
10 Jun 2025
On Finetuning Tabular Foundation Models
Ivan Rubachev
Akim Kotelnikov
Nikolay Kartashev
Artem Babenko
29
0
0
10 Jun 2025
Overclocking LLM Reasoning: Monitoring and Controlling Thinking Path Lengths in LLMs
Overclocking LLM Reasoning: Monitoring and Controlling Thinking Path Lengths in LLMs
Roy Eisenstadt
Itamar Zimerman
Lior Wolf
LRM
15
0
0
08 Jun 2025
Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction
Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction
Zesheng Ye
C. Cai
Ruijiang Dong
Jianzhong Qi
Lei Feng
Pin-Yu Chen
Feng Liu
208
0
0
05 Jun 2025
Sample Complexity and Representation Ability of Test-time Scaling Paradigms
Sample Complexity and Representation Ability of Test-time Scaling Paradigms
Baihe Huang
Shanda Li
Tianhao Wu
Yiming Yang
Ameet Talwalkar
Kannan Ramchandran
Michael I. Jordan
Jiantao Jiao
LRM
109
0
0
05 Jun 2025
A Statistical Physics of Language Model Reasoning
Jack David Carson
Amir Reisizadeh
LRMAI4CE
78
0
0
04 Jun 2025
SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling
SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling
Anhao Zhao
Fanghua Ye
Yingqi Fan
Junlong Tong
Zhiwei Fei
Hui Su
Xiaoyu Shen
66
0
0
04 Jun 2025
Bridging Neural ODE and ResNet: A Formal Error Bound for Safety Verification
Bridging Neural ODE and ResNet: A Formal Error Bound for Safety Verification
Abdelrahman Sayed Sayed
Pierre-Jean Meyer
Mohamed Ghazel
29
0
0
03 Jun 2025
The Unified Cognitive Consciousness Theory for Language Models: Anchoring Semantics, Thresholds of Activation, and Emergent Reasoning
The Unified Cognitive Consciousness Theory for Language Models: Anchoring Semantics, Thresholds of Activation, and Emergent Reasoning
Edward Y. Chang
LRM
14
0
0
02 Jun 2025
Attention Retrieves, MLP Memorizes: Disentangling Trainable Components in the Transformer
Attention Retrieves, MLP Memorizes: Disentangling Trainable Components in the Transformer
Yihe Dong
Lorenzo Noci
Mikhail Khodak
Mufan Li
43
0
0
01 Jun 2025
Circuit Stability Characterizes Language Model Generalization
Circuit Stability Characterizes Language Model Generalization
Alan Sun
LRM
22
0
0
30 May 2025
Characterising the Inductive Biases of Neural Networks on Boolean Data
Characterising the Inductive Biases of Neural Networks on Boolean Data
Chris Mingard
Lukas Seier
Niclas Goring
Andrei-Vlad Badelita
Charles London
Ard A. Louis
AI4CE
37
0
0
29 May 2025
LoLA: Low-Rank Linear Attention With Sparse Caching
LoLA: Low-Rank Linear Attention With Sparse Caching
Luke McDermott
Robert W. Heath Jr.
Rahul Parhi
RALM
53
0
0
29 May 2025
Geometric Hyena Networks for Large-scale Equivariant Learning
Geometric Hyena Networks for Large-scale Equivariant Learning
Artem Moskalev
Mangal Prakash
Junjie Xu
Tianyu Cui
Rui Liao
Tommaso Mansi
45
1
0
28 May 2025
Born a Transformer -- Always a Transformer?
Born a Transformer -- Always a Transformer?
Yana Veitsman
Mayank Jobanputra
Yash Sarrof
Aleksandra Bakalova
Vera Demberg
Ellie Pavlick
Michael Hahn
61
0
0
27 May 2025
Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspective
Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspective
Junnan Liu
Hongwei Liu
Linchen Xiao
Shudong Liu
Taolin Zhang
Zihan Ma
Songyang Zhang
Kai Chen
LRM
120
0
0
26 May 2025
How Syntax Specialization Emerges in Language Models
How Syntax Specialization Emerges in Language Models
Xufeng Duan
Zhaoqian Yao
Yunhao Zhang
Shaonan Wang
Zhenguang G. Cai
MILMLRM
71
0
0
26 May 2025
The Coverage Principle: A Framework for Understanding Compositional Generalization
The Coverage Principle: A Framework for Understanding Compositional Generalization
Hoyeon Chang
Jinho Park
Hanseul Cho
Sohee Yang
Miyoung Ko
Hyeonbin Hwang
Seungpil Won
Dohaeng Lee
Youbin Ahn
Minjoon Seo
59
0
0
26 May 2025
The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models
The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models
Shashata Sawmya
Micah Adler
Nir Shavit
MILM
31
0
0
26 May 2025
Understanding Transformer from the Perspective of Associative Memory
Understanding Transformer from the Perspective of Associative Memory
Shu Zhong
Mingyu Xu
Tenglong Ao
Guang Shi
47
1
0
26 May 2025
Dynamic Manifold Evolution Theory: Modeling and Stability Analysis of Latent Representations in Large Language Models
Dynamic Manifold Evolution Theory: Modeling and Stability Analysis of Latent Representations in Large Language Models
Yukun Zhang
Qi Dong
AI4CE
21
0
0
24 May 2025
TRACE for Tracking the Emergence of Semantic Representations in Transformers
TRACE for Tracking the Emergence of Semantic Representations in Transformers
Nura Aljaafari
Danilo S. Carvalho
André Freitas
92
0
0
23 May 2025
The emergence of sparse attention: impact of data distribution and benefits of repetition
The emergence of sparse attention: impact of data distribution and benefits of repetition
Nicolas Zucchet
Francesco dÁngelo
Andrew Kyle Lampinen
Stephanie C. Y. Chan
214
1
0
23 May 2025
Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models
Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models
Patrick Leask
Neel Nanda
Noura Al Moubayed
87
1
0
23 May 2025
Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence
Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence
Gouki Minegishi
Hiroki Furuta
Shohei Taniguchi
Yusuke Iwasawa
Yutaka Matsuo
81
0
0
22 May 2025
On the creation of narrow AI: hierarchy and nonlocality of neural network skills
On the creation of narrow AI: hierarchy and nonlocality of neural network skills
Eric J. Michaud
Asher Parker-Sartori
Max Tegmark
109
0
0
21 May 2025
The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models
The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models
Adrian Cosma
Stefan Ruseti
Emilian Radoi
Mihai Dascalu
LRM
75
0
0
20 May 2025
Do Language Models Use Their Depth Efficiently?
Do Language Models Use Their Depth Efficiently?
Róbert Csordás
Christopher D. Manning
Christopher Potts
208
2
0
20 May 2025
Soft Prompts for Evaluation: Measuring Conditional Distance of Capabilities
Soft Prompts for Evaluation: Measuring Conditional Distance of Capabilities
Ross Nordby
58
0
0
20 May 2025
Language Models use Lookbacks to Track Beliefs
Language Models use Lookbacks to Track Beliefs
Nikhil Prakash
Natalie Shapira
Arnab Sen Sharma
Christoph Riedl
Yonatan Belinkov
Tamar Rott Shaham
David Bau
Atticus Geiger
KELM
82
1
0
20 May 2025
Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs
Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs
Zhipeng Yang
Junzhuo Li
Siyu Xia
Xuming Hu
AIFinLRM
105
0
0
20 May 2025
AutoMathKG: The automated mathematical knowledge graph based on LLM and vector database
AutoMathKG: The automated mathematical knowledge graph based on LLM and vector database
Rong Bian
Yu Geng
Zijian Yang
Bing Cheng
124
0
0
19 May 2025
Panda: A pretrained forecast model for universal representation of chaotic dynamics
Panda: A pretrained forecast model for universal representation of chaotic dynamics
Jeffrey Lai
Anthony Bao
William Gilpin
AI4TSAI4CE
101
0
0
19 May 2025
Understanding Task Representations in Neural Networks via Bayesian Ablation
Understanding Task Representations in Neural Networks via Bayesian Ablation
Andrew Nam
Declan Campbell
Thomas Griffiths
Jonathan Cohen
Sarah-Jane Leslie
BDL
40
0
0
19 May 2025
Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models
Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models
Shuxun Wang
Qingyu Yin
Chak Tou Leong
Qiang Zhang
Linyi Yang
58
0
0
17 May 2025
Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors
Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors
Jing Huang
Junyi Tao
Thomas Icard
Diyi Yang
Christopher Potts
OODD
87
0
0
17 May 2025
Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction
Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction
Jeffrey Willette
Heejun Lee
Sung Ju Hwang
73
0
0
16 May 2025
Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning
Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning
Jingcheng Niu
Subhabrata Dutta
Ahmed Elshabrawy
Harish Tayyar Madabushi
Iryna Gurevych
148
1
0
16 May 2025
Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs
Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs
Jingcheng Niu
Xingdi Yuan
Tong Wang
Hamidreza Saghir
Amir H. Abdi
73
0
0
14 May 2025
AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection
AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection
Kai Hua
Steven Wu
Ge Zhang
Ke Shen
LRM
83
0
0
12 May 2025
Task Reconstruction and Extrapolation for $π_0$ using Text Latent
Task Reconstruction and Extrapolation for π0π_0 using Text Latent
Quanyi Li
103
0
0
06 May 2025
Quiet Feature Learning in Algorithmic Tasks
Quiet Feature Learning in Algorithmic Tasks
Prudhviraj Naidu
Zixian Wang
Leon Bergen
R. Paturi
VLM
113
0
0
06 May 2025
On the generalization of language models from in-context learning and finetuning: a controlled study
On the generalization of language models from in-context learning and finetuning: a controlled study
Andrew Kyle Lampinen
Arslan Chaudhry
Stephanie Chan
Cody Wild
Diane Wan
Alex Ku
Jorg Bornschein
Razvan Pascanu
Murray Shanahan
James L. McClelland
169
5
0
01 May 2025

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.