v1v2 (latest)

Do Language Models Use Their Depth Efficiently?

20 May 2025

Róbert Csordás

Christopher D. Manning

Christopher Potts

ArXiv (abs)PDF HTML

Papers citing "Do Language Models Use Their Depth Efficiently?"

35 / 35 papers shown

Title
Transferring Features Across Language Models With Model Stitching Alan Chen Jack Merullo Alessandro Stolfo Ellie Pavlick 103 0 0 07 Jun 2025
Layer by Layer: Uncovering Hidden Representations in Language Models Oscar Skean Md Rifat Arefin Dan Zhao Niket Patel Jalal Naghiyev Yann LeCun Ravid Shwartz-Ziv MILM AIFin 333 52 0 04 Feb 2025
NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals Jaden Fiotto-Kaufman Alexander R. Loftus Eric Todd Jannik Brinkmann Caden Juang ... Carla Brodley Arjun Guha Jonathan Bell Byron C. Wallace David Bau 156 9 0 18 Jul 2024
Transformer Layers as Painters Qi Sun Marc Pickett Aakash Kumar Nain Llion Jones AI4CE 216 26 0 12 Jul 2024
The Remarkable Robustness of LLMs: Stages of Inference? Vedang Lad Wes Gurnee Max Tegmark Max Tegmark 175 64 0 27 Jun 2024
MoEUT: Mixture-of-Experts Universal Transformers Róbert Csordás Kazuki Irie Jürgen Schmidhuber Christopher Potts Christopher D. Manning MoE 118 14 0 25 May 2024
The Unreasonable Ineffectiveness of the Deeper Layers Andrey Gromov Kushal Tirumala Hassan Shapourian Paolo Glorioso Daniel A. Roberts 201 125 0 26 Mar 2024
RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations Jing-ling Huang Zhengxuan Wu Christopher Potts Mor Geva Atticus Geiger 172 41 0 27 Feb 2024
Successor Heads: Recurring, Interpretable Attention Heads In The Wild Rhys Gould Euan Ong George Ogden Arthur Conmy LRM 86 56 0 14 Dec 2023
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time Zichang Liu Jue Wang Tri Dao Dinesh Manocha Binhang Yuan ... Anshumali Shrivastava Ce Zhang Yuandong Tian Christopher Ré Beidi Chen BDL 171 239 0 26 Oct 2023
Copy Suppression: Comprehensively Understanding an Attention Head Callum McDougall Arthur Conmy Cody Rushing Thomas McGrath Neel Nanda MILM 97 48 0 06 Oct 2023
Language Models Represent Space and Time Wes Gurnee Max Tegmark 203 194 0 03 Oct 2023
Neurons in Large Language Models: Dead, N-gram, Positional Elena Voita Javier Ferrando Christoforos Nalmpantis MILM 204 62 0 09 Sep 2023
MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions Zexuan Zhong Zhengxuan Wu Christopher D. Manning Christopher Potts Danqi Chen KELM 204 235 0 24 May 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.8K 16,569 0 15 Mar 2023
Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations Atticus Geiger Zhengxuan Wu Christopher Potts Thomas Icard Noah D. Goodman CML 238 124 0 05 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 1.9K 14,889 0 27 Feb 2023
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 422 586 0 24 Sep 2022
Toy Models of Superposition Nelson Elhage Tristan Hume Catherine Olsson Nicholas Schiefer T. Henighan ... Sam McCandlish Jared Kaplan Dario Amodei Martin Wattenberg C. Olah AAML MILM 304 425 0 21 Sep 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 1.4K 11,179 0 28 Jan 2022
Training Verifiers to Solve Math Word Problems K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun ... Jerry Tworek Jacob Hilton Reiichiro Nakano Christopher Hesse John Schulman ReLM OffRL LRM 533 5,201 0 27 Oct 2021
PonderNet: Learning to Ponder Andrea Banino Jan Balaguer Charles Blundell PINN AIMat 217 88 0 12 Jul 2021
Measuring Mathematical Problem Solving With the MATH Dataset Dan Hendrycks Collin Burns Saurav Kadavath Akul Arora Steven Basart Eric Tang Basel Alomair Jacob Steinhardt ReLM FaML 310 2,857 0 05 Mar 2021
Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks Róbert Csordás Sjoerd van Steenkiste Jürgen Schmidhuber 174 100 0 05 Oct 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 1.5K 45,384 0 28 May 2020
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection Shauli Ravfogel Yanai Elazar Hila Gonen Michael Twiton Yoav Goldberg 259 408 0 16 Apr 2020
On Layer Normalization in the Transformer Architecture Ruibin Xiong Yunchang Yang Di He Kai Zheng Shuxin Zheng Chen Xing Huishuai Zhang Yanyan Lan Liwei Wang Tie-Yan Liu AI4CE 185 1,065 0 12 Feb 2020
Root Mean Square Layer Normalization Biao Zhang Rico Sennrich 321 888 0 16 Oct 2019
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings Kawin Ethayarajh 163 907 0 02 Sep 2019
Analysing Mathematical Reasoning Abilities of Neural Models D. Saxton Edward Grefenstette Felix Hill Pushmeet Kohli LRM 335 446 0 02 Apr 2019
Universal Transformers Mostafa Dehghani Stephan Gouws Oriol Vinyals Jakob Uszkoreit Lukasz Kaiser 189 774 0 10 Jul 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 1.5K 140,362 0 12 Jun 2017
Axiomatic Attribution for Deep Networks Mukund Sundararajan Ankur Taly Qiqi Yan OOD FAtt 507 6,321 0 04 Mar 2017
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 473 10,848 0 21 Jul 2016
Adaptive Computation Time for Recurrent Neural Networks Alex Graves 264 571 0 29 Mar 2016