v1v2v3v4v5 (latest)

Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs

13 September 2023

Papers citing "Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs"

41 / 41 papers shown

Title
Shaping Shared Languages: Human and Large Language Models' Inductive Biases in Emergent Communication Tom Kouwenhoven Max Peeperkorn R. D. Kleijn Tessa Verhoef 101 0 0 06 Mar 2025
Mechanistic Interpretability of Emotion Inference in Large Language Models Ala Nekouvaght Tak Amin Banayeeanzade Anahita Bolourani Mina Kian Robin Jia Jonathan Gratch 102 0 0 08 Feb 2025
Geometric Signatures of Compositionality Across a Language Model's Lifetime Jin Hwa Lee Thomas Jiralerspong Lei Yu Yoshua Bengio Emily Cheng CoGe 182 4 0 02 Oct 2024
A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models Daking Rai Yilun Zhou Shi Feng Abulhair Saparov Ziyu Yao 176 33 0 02 Jul 2024
Natural Language Processing RELIES on Linguistics Juri Opitz Shira Wein Nathan Schneider AI4CE 137 8 0 09 May 2024
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models Samuel Marks Can Rager Eric J. Michaud Yonatan Belinkov David Bau Aaron Mueller 140 159 0 28 Mar 2024
Latent State Models of Training Dynamics Michael Y. Hu Angelica Chen Naomi Saphra Kyunghyun Cho 76 8 0 18 Aug 2023
On the special role of class-selective neurons in early training Omkar Ranadive Nikhil Thakurdesai Ari S. Morcos Matthew L. Leavitt Stéphane Deny 64 2 0 27 May 2023
A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks William Merrill Nikolaos Tsilivis Aman Shukla 67 54 0 21 Mar 2023
Characterizing Intrinsic Compositionality in Transformers with Tree Projections Shikhar Murty Pratyusha Sharma Jacob Andreas Christopher D. Manning 87 40 0 02 Nov 2022
Measuring and Narrowing the Compositionality Gap in Language Models Ofir Press Muru Zhang Sewon Min Ludwig Schmidt Noah A. Smith M. Lewis ReLM KELM LRM 202 643 0 07 Oct 2022
Omnigrok: Grokking Beyond Algorithmic Data Ziming Liu Eric J. Michaud Max Tegmark 93 84 0 03 Oct 2022
Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit Boaz Barak Benjamin L. Edelman Surbhi Goel Sham Kakade Eran Malach Cyril Zhang 108 133 0 18 Jul 2022
Emergent Abilities of Large Language Models Jason W. Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph ... Tatsunori Hashimoto Oriol Vinyals Percy Liang J. Dean W. Fedus ELM ReLM LRM 292 2,521 0 15 Jun 2022
Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation Verna Dankers Christopher G. Lucas Ivan Titov 80 38 0 30 May 2022
Locating and Editing Factual Associations in GPT Kevin Meng David Bau A. Andonian Yonatan Belinkov KELM 251 1,389 0 10 Feb 2022
Acquisition of Chess Knowledge in AlphaZero Thomas McGrath A. Kapishnikov Nenad Tomašev Adam Pearce Demis Hassabis Been Kim Ulrich Paquet Vladimir Kramnik 62 168 0 17 Nov 2021
The Grammar-Learning Trajectories of Neural Language Models Leshem Choshen Guy Hacohen D. Weinshall Omri Abend 97 29 0 13 Sep 2021
Datasets: A Community Library for Natural Language Processing Quentin Lhoest Albert Villanova del Moral Yacine Jernite A. Thakur Patrick von Platen ... Thibault Goehringer Victor Mustar François Lagunas Alexander M. Rush Thomas Wolf 221 614 0 07 Sep 2021
Attention Flows are Shapley Value Explanations Kawin Ethayarajh Dan Jurafsky FAtt TDI 69 35 0 31 May 2021
What if This Modified That? Syntactic Interventions via Counterfactual Embeddings Mycal Tucker Peng Qian R. Levy 62 40 0 28 May 2021
Probing Across Time: What Does RoBERTa Know and When? Leo Z. Liu Yizhong Wang Jungo Kasai Hannaneh Hajishirzi Noah A. Smith KELM 100 88 0 16 Apr 2021
Gradient Starvation: A Learning Proclivity in Neural Networks Mohammad Pezeshki Sekouba Kaba Yoshua Bengio Aaron Courville Doina Precup Guillaume Lajoie MLT 136 269 0 18 Nov 2020
Pareto Probing: Trading Off Accuracy for Complexity Tiago Pimentel Naomi Saphra Adina Williams Ryan Cotterell 71 60 0 05 Oct 2020
The Pitfalls of Simplicity Bias in Neural Networks Harshay Shah Kaustav Tamuly Aditi Raghunathan Prateek Jain Praneeth Netrapalli AAML 72 364 0 13 Jun 2020
Quantifying Attention Flow in Transformers Samira Abnar Willem H. Zuidema 169 803 0 02 May 2020
The Break-Even Point on Optimization Trajectories of Deep Neural Networks Stanislaw Jastrzebski Maciej Szymczak Stanislav Fort Devansh Arpit Jacek Tabor Kyunghyun Cho Krzysztof J. Geras 88 164 0 21 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 651 4,925 0 23 Jan 2020
BLiMP: The Benchmark of Linguistic Minimal Pairs for English Alex Warstadt Alicia Parrish Haokun Liu Anhad Mohananey Wei Peng Sheng-Fu Wang Samuel R. Bowman 100 495 0 02 Dec 2019
What Does BERT Look At? An Analysis of BERT's Attention Kevin Clark Urvashi Khandelwal Omer Levy Christopher D. Manning MILM 235 1,605 0 11 Jun 2019
Is Attention Interpretable? Sofia Serrano Noah A. Smith 110 686 0 09 Jun 2019
Intrinsic dimension of data representations in deep neural networks A. Ansuini Alessandro Laio Jakob H. Macke D. Zoccolan AI4CE 90 279 0 29 May 2019
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned Elena Voita David Talbot F. Moiseev Rico Sennrich Ivan Titov 119 1,149 0 23 May 2019
Similarity of Neural Network Representations Revisited Simon Kornblith Mohammad Norouzi Honglak Lee Geoffrey E. Hinton 143 1,436 0 01 May 2019
The Scientific Method in the Science of Machine Learning Jessica Zosa Forde Michela Paganini 70 37 0 24 Apr 2019
Attention is not Explanation Sarthak Jain Byron C. Wallace FAtt 152 1,330 0 26 Feb 2019
Deep learning generalizes because the parameter-function map is biased towards simple functions Guillermo Valle Pérez Chico Q. Camargo A. Louis MLT AI4CE 113 232 0 22 May 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 1.1K 7,201 0 20 Apr 2018
Estimating the intrinsic dimension of datasets by a minimal neighborhood information Elena Facco M. d’Errico Alex Rodriguez Alessandro Laio 65 327 0 19 Mar 2018
A Closer Look at Memorization in Deep Networks Devansh Arpit Stanislaw Jastrzebski Nicolas Ballas David M. Krueger Emmanuel Bengio ... Tegan Maharaj Asja Fischer Aaron Courville Yoshua Bengio Simon Lacoste-Julien TDI 136 1,829 0 16 Jun 2017
Neural Machine Translation of Rare Words with Subword Units Rico Sennrich Barry Haddow Alexandra Birch 238 7,765 0 31 Aug 2015