Title
N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs Ilya Zisman Alexander Nikulin Andrei Polubarov Nikita Lyubaykin Vladislav Kurenkov Andrei Polubarov Igor Kiselev Vladislav Kurenkov OffRL 136 2 0 04 Nov 2024
A Mechanistic Explanatory Strategy for XAI Marcin Rabiza 140 1 0 02 Nov 2024
Toward Understanding In-context vs. In-weight Learning Bryan Chan Xinyi Chen András Gyorgy Dale Schuurmans 155 5 0 30 Oct 2024
Abrupt Learning in Transformers: A Case Study on Matrix Completion Pulkit Gopalani Ekdeep Singh Lubana Wei Hu 73 4 0 29 Oct 2024
Understanding Synthetic Context Extension via Retrieval Heads Xinyu Zhao Fangcong Yin Greg Durrett 141 2 0 29 Oct 2024
Causal Interventions on Causal Paths: Mapping GPT-2's Reasoning From Syntax to Semantics Isabelle Lee Joshua Lum Ziyi Liu Dani Yogatama LRM 60 0 0 28 Oct 2024
DAWN-ICL: Strategic Planning of Problem-solving Trajectories for Zero-Shot In-Context Learning Xinyu Tang Xiaolei Wang Wayne Xin Zhao Ji-Rong Wen 121 6 0 26 Oct 2024
Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination Jerry Huang Prasanna Parthasarathi Mehdi Rezagholizadeh Boxing Chen Sarath Chandar 169 0 0 22 Oct 2024
Revealing and Mitigating the Local Pattern Shortcuts of Mamba Wangjie You Zecheng Tang Juntao Li Lili Yao Min Zhang Mamba 61 0 0 21 Oct 2024
Opportunities and Challenges of Generative-AI in Finance Akshar Prabhu Desai Ganesh Satish Mallya Mohammad Luqman Tejasvi Ravi Nithya Kota Pranjul Yadav AIFin 125 4 0 21 Oct 2024
How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs Guhao Feng Kai-Bo Yang Yuntian Gu Xinyue Ai Shengjie Luo Jiacheng Sun Di He Zechao Li Liwei Wang LRM 85 13 0 17 Oct 2024
In-context learning and Occam's razor Eric Elmoznino Tom Marty Tejas Kasetty Léo Gagnon Sarthak Mittal Mahan Fathi Dhanya Sridhar Guillaume Lajoie 162 1 0 17 Oct 2024
Hypothesis Testing the Circuit Hypothesis in LLMs Claudia Shi Nicolas Beltran-Velez Achille Nazaret Carolina Zheng Adrià Garriga-Alonso Andrew Jesson Maggie Makar David M. Blei 100 12 0 16 Oct 2024
Mimetic Initialization Helps State Space Models Learn to Recall Asher Trockman Hrayr Harutyunyan J. Zico Kolter Sanjiv Kumar Srinadh Bhojanapalli Mamba 47 5 0 14 Oct 2024
Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts Guorui Zheng Xidong Wang Juhao Liang Nuo Chen Yuping Zheng Benyou Wang MoE 124 5 0 14 Oct 2024
HSR-Enhanced Sparse Attention Acceleration Bo Chen Yingyu Liang Zhizhou Sha Zhenmei Shi Zhao Song 261 22 0 14 Oct 2024
Fine-grained Attention I/O Complexity: Comprehensive Analysis for Backward Passes Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao Song Yufa Zhou 94 18 0 12 Oct 2024
MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models Jiachun Li Pengfei Cao Zhuoran Jin Yubo Chen Kang Liu Jun Zhao LRM ELM 80 7 0 12 Oct 2024
Inference and Verbalization Functions During In-Context Learning Junyi Tao Xiaoyin Chen Nelson F. Liu LRM ReLM 90 1 0 12 Oct 2024
The Same But Different: Structural Similarities and Differences in Multilingual Language Modeling Ruochen Zhang Qinan Yu Matianyu Zang Carsten Eickhoff Ellie Pavlick 91 6 0 11 Oct 2024
SAGE: Scalable Ground Truth Evaluations for Large Sparse Autoencoders Constantin Venhoff Anisoara Calinescu Philip Torr Christian Schroeder de Witt 74 0 0 09 Oct 2024
Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures Junxuan Wang Xuyang Ge Wentao Shu Qiong Tang Yunhua Zhou Zhengfu He Xipeng Qiu 82 7 0 09 Oct 2024
Transformers learn variable-order Markov chains in-context Ruida Zhou C. Tian Suhas Diggavi 78 1 0 07 Oct 2024
Task Diversity Shortens the ICL Plateau Jaeyeon Kim Sehyun Kwon Joo Young Choi Jongho Park Jaewoong Cho Jason D. Lee Ernest K. Ryu MoMe 99 3 0 07 Oct 2024
RespDiff: An End-to-End Multi-scale RNN Diffusion Model for Respiratory Waveform Estimation from PPG Signals Yuyang Miao Zehua Chen Chong Li Danilo Mandic DiffM MedIm 79 9 0 06 Oct 2024
In-context Learning in Presence of Spurious Correlations Hrayr Harutyunyan R. Darbinyan Samvel Karapetyan Hrant Khachatrian LRM 88 1 0 04 Oct 2024
Can Transformers Learn $n$ -gram Language Models? Anej Svete Nadav Borenstein M. Zhou Isabelle Augenstein Ryan Cotterell 99 8 0 03 Oct 2024
Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient George Wang Jesse Hoogland Stan van Wingerden Zach Furman Daniel Murfet OffRL 86 9 0 03 Oct 2024
Listening to the Wise Few: Select-and-Copy Attention Heads for Multiple-Choice QA Eduard Tulchinskii Laida Kushnareva Kristian Kuznetsov Anastasia Voznyuk Andrei Andriiainen Irina Piontkovskaya Evgeny Burnaev Serguei Barannikov 123 1 0 03 Oct 2024
Softmax is not Enough (for Sharp Size Generalisation) Petar Velickovic Christos Perivolaropoulos Federico Barbero Razvan Pascanu 114 17 0 01 Oct 2024
PEAR: Position-Embedding-Agnostic Attention Re-weighting Enhances Retrieval-Augmented Generation with Zero Inference Overhead Tao Tan Yining Qian Ang Lv Hongzhan Lin Songhao Wu Yongbo Wang Feng Wang Jingtong Wu Xin Lu Rui Yan 85 1 0 29 Sep 2024
Neural Decompiling of Tracr Transformers Hannes Thurnherr Kaspar Riesen ViT 48 2 0 29 Sep 2024
On the Inductive Bias of Stacking Towards Improving Reasoning Nikunj Saunshi Stefani Karp Shankar Krishnan Sobhan Miryoosefi Sashank J. Reddi Sanjiv Kumar LRM AI4CE 86 7 0 27 Sep 2024
In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Models Pengrui Han Peiyang Song Haofei Yu Jiaxuan You ReLM LRM 76 1 0 23 Sep 2024
Optimal ablation for interpretability Maximilian Li Lucas Janson FAtt 112 3 0 16 Sep 2024
Flash STU: Fast Spectral Transform Units Y. Isabel Liu Windsor Nguyen Yagiz Devre Evan Dogariu Anirudha Majumdar Elad Hazan AI4TS 156 1 0 16 Sep 2024
Language Models "Grok" to Copy Ang Lv Ruobing Xie Xingwu Sun Zhanhui Kang Rui Yan LLMAG 144 0 0 14 Sep 2024
Attention Heads of Large Language Models: A Survey Zifan Zheng Yezhaohui Wang Yuxin Huang Shichao Song Mingchuan Yang Bo Tang Feiyu Xiong Zhiyu Li LRM 117 29 0 05 Sep 2024
Beyond Model Interpretability: Socio-Structural Explanations in Machine Learning Andrew Smart Atoosa Kasirzadeh 96 6 0 05 Sep 2024
Masked Mixers for Language Generation and Retrieval Benjamin L. Badger 167 0 0 02 Sep 2024
Investigating Neuron Ablation in Attention Heads: The Case for Peak Activation Centering Nicholas Pochinkov Ben Pasero Skylar Shibayama 67 2 0 30 Aug 2024
One-layer transformers fail to solve the induction heads task Clayton Sanford Daniel J. Hsu Matus Telgarsky 106 12 0 26 Aug 2024
Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neuroscience Zhonghao He Jascha Achterberg Katie Collins Kevin K. Nejad Danyal Akarca ... Chole Li Kai J. Sandbrink Stephen Casper Anna Ivanova Grace W. Lindsay AI4CE 99 2 0 22 Aug 2024
In-Context Learning with Representations: Contextual Generalization of Trained Transformers Tong Yang Yu Huang Yingbin Liang Yuejie Chi MLT 115 12 0 19 Aug 2024
Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference Geonhee Kim Marco Valentino André Freitas LRM AI4CE 106 11 0 16 Aug 2024
Large Language Models Might Not Care What You Are Saying: Prompt Format Beats Descriptions Chenming Tang Zhixiang Wang Hao Sun Yunfang Wu LRM 115 0 0 16 Aug 2024
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 Tom Lieberum Senthooran Rajamanoharan Arthur Conmy Lewis Smith Nicolas Sonnerat Vikrant Varma János Kramár Anca Dragan Rohin Shah Neel Nanda 121 128 0 09 Aug 2024
The Quest for the Right Mediator: A History, Survey, and Theoretical Grounding of Causal Interpretability Aaron Mueller Jannik Brinkmann Millicent Li Samuel Marks Koyena Pal ... Arnab Sen Sharma Jiuding Sun Eric Todd David Bau Yonatan Belinkov CML 130 25 0 02 Aug 2024
Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability Jorge García-Carrasco A. Maté Juan Trujillo AAML 77 3 0 29 Jul 2024
Efficient LLM Training and Serving with Heterogeneous Context Sharding among Attention Heads Xihui Lin Yunan Zhang Suyu Ge Barun Patra Vishrav Chaudhary Hao Peng Xia Song 48 0 0 25 Jul 2024