Born a Transformer -- Always a Transformer?

v1v2 (latest)

Born a Transformer -- Always a Transformer?

27 May 2025

Mayank Jobanputra

Aleksandra Bakalova

ArXiv (abs)PDF HTML

Papers citing "Born a Transformer -- Always a Transformer?"

6 / 6 papers shown

Title
Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B Aleksandra Bakalova Yana Veitsman Xinting Huang Michael Hahn 76 2 0 31 Mar 2025
Supposedly Equivalent Facts That Aren't? Entity Frequency in Pre-training Induces Asymmetry in LLMs Yuan He Bailan He Zifeng Ding Alisia Lupidi Yuqicheng Zhu ... Caiqi Zhang Jiaoyan Chen Yunpu Ma Volker Tresp Ian Horrocks 89 2 0 28 Mar 2025
Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Tracking Yifan Zhang Wenyu Du Dongming Jin Jie Fu Zhi Jin LRM 132 2 0 27 Feb 2025
Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers Alireza Amiri Xinting Huang Mark Rofin Michael Hahn LRM 575 3 0 04 Feb 2025
Out-of-distribution generalization via composition: a lens through induction heads in Transformers Jiajun Song Zhuoyan Xu Yiqiao Zhong 158 10 0 31 Dec 2024
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues Riccardo Grazzi Julien N. Siems Jörg Franke Arber Zela Frank Hutter Massimiliano Pontil 210 26 0 19 Nov 2024