Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

6 January 2022

Papers citing "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets"

50 / 86 papers shown

Title
Distilled Circuits: A Mechanistic Study of Internal Restructuring in Knowledge Distillation Reilly Haskins Benjamin Adams 16 0 0 16 May 2025
Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis Akarsh Kumar Jeff Clune Joel Lehman Kenneth O. Stanley OOD 21 0 0 16 May 2025
Quiet Feature Learning in Algorithmic Tasks Prudhviraj Naidu Zixian Wang Leon Bergen R. Paturi VLM 57 0 0 06 May 2025
Reinforcement Learning for Reasoning in Large Language Models with One Training Example Yiping Wang Qing Yang Zhiyuan Zeng Liliang Ren L. Liu ... Jianfeng Gao Weizhu Chen S. Wang Simon S. Du Yelong Shen OffRL ReLM LRM 131 9 0 29 Apr 2025
Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformers Roman Abramov Felix Steinbauer Gjergji Kasneci 207 0 0 29 Apr 2025
NeuralGrok: Accelerate Grokking by Neural Gradient Transformation Xinyu Zhou Simin Fan Martin Jaggi Jie Fu 41 0 0 24 Apr 2025
Beyond the Next Token: Towards Prompt-Robust Zero-Shot Classification via Efficient Multi-Token Prediction Junlang Qian Zixiao Zhu Hanzhang Zhou Zijian Feng Zepeng Zhai K. Mao AAML VLM 43 0 0 04 Apr 2025
Low Rank and Sparse Fourier Structure in Recurrent Networks Trained on Modular Addition Akshay Rangamani 45 0 0 28 Mar 2025
From Small to Large Language Models: Revisiting the Federalist Papers So Won Jeong Veronika Rockova 42 0 0 25 Feb 2025
Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs Jan Betley Daniel Tan Niels Warncke Anna Sztyber-Betley Xuchan Bao Martín Soto Nathan Labenz Owain Evans AAML 80 12 0 24 Feb 2025
Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations Yize Zhao Tina Behnia V. Vakilian Christos Thrampoulidis 70 9 0 20 Feb 2025
Early Stopping Against Label Noise Without Validation Data Suqin Yuan Lei Feng Tongliang Liu NoLa 104 17 0 11 Feb 2025
Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries Chris Kolb T. Weber Bernd Bischl David Rügamer 115 0 0 04 Feb 2025
Grokking Explained: A Statistical Phenomenon B. W. Carvalho Artur Garcez Luís C. Lamb Emílio Vital Brazil 69 0 0 03 Feb 2025
Harmonic Loss Trains Interpretable AI Models David D. Baek Ziming Liu Riya Tyagi Max Tegmark 97 2 0 03 Feb 2025
It's Not Just a Phase: On Investigating Phase Transitions in Deep Learning-based Side-channel Analysis Sengim Karayalçin Marina Krček Stjepan Picek AAML 80 0 0 01 Feb 2025
Physics of Skill Learning Ziming Liu Yizhou Liu Eric J. Michaud Jeff Gore Max Tegmark 54 2 0 21 Jan 2025
Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words Gouki Minegishi Hiroki Furuta Yusuke Iwasawa Y. Matsuo 53 1 0 09 Jan 2025
Grokking at the Edge of Numerical Stability Lucas Prieto Melih Barsbey Pedro A.M. Mediano Tolga Birdal 51 3 0 08 Jan 2025
An Analysis Framework for Understanding Deep Neural Networks Based on Network Dynamics Yuchen Lin Yong Zhang Sihan Feng Hong Zhao 41 0 0 05 Jan 2025
How to explain grokking S. V. Kozyrev AI4CE 36 0 0 03 Jan 2025
Out-of-distribution generalization via composition: a lens through induction heads in Transformers Jiajun Song Zhuoyan Xu Yiqiao Zhong 88 4 0 31 Dec 2024
On Memorization of Large Language Models in Logical Reasoning Chulin Xie Yangsibo Huang Chiyuan Zhang Da Yu Xinyun Chen Bill Yuchen Lin Bo Li Badih Ghazi Ravi Kumar LRM 58 24 0 30 Oct 2024
Chemistry-Inspired Diffusion with Non-Differentiable Guidance Yuchen Shen Chenhao Zhang Sijie Fu Chenghui Zhou Newell Washburn Barnabás Póczos 60 0 0 09 Oct 2024
Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training Kun Song Zhiquan Tan Bochao Zou Jiansheng Chen Huimin Ma Weiran Huang 44 1 0 25 Sep 2024
Zero-shot forecasting of chaotic systems Yuanzhao Zhang William Gilpin AI4TS 42 6 0 24 Sep 2024
Language Models "Grok" to Copy Ang Lv Ruobing Xie Xingwu Sun Zhanhui Kang Rui Yan LLMAG 53 2 0 14 Sep 2024
On the Geometry of Deep Learning Randall Balestriero Ahmed Imtiaz Humayun Richard G. Baraniuk AI4CE 47 1 0 09 Aug 2024
A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models Daking Rai Yilun Zhou Shi Feng Abulhair Saparov Ziyu Yao 85 22 0 02 Jul 2024
How Do Large Language Models Acquire Factual Knowledge During Pretraining? Hoyeon Chang Jinho Park Seonghyeon Ye Sohee Yang Youngkyung Seo Du-Seong Chang Minjoon Seo KELM 37 33 0 17 Jun 2024
Bayesian RG Flow in Neural Network Field Theories Jessica N. Howard Marc S. Klinger Anindita Maiti A. G. Stapleton 68 1 0 27 May 2024
Unsupervised Meta-Learning via In-Context Learning Anna Vettoruzzo Lorenzo Braccaioli Joaquin Vanschoren M. Nowaczyk SSL 72 0 0 25 May 2024
Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory Xueyan Niu Bo Bai Lei Deng Wei Han 44 6 0 14 May 2024
Rho-1: Not All Tokens Are What You Need Zheng-Wen Lin Zhibin Gou Yeyun Gong Xiao Liu Yelong Shen ... Chen Lin Yujiu Yang Jian Jiao Nan Duan Weizhu Chen CLL 50 57 0 11 Apr 2024
Understanding Emergent Abilities of Language Models from the Loss Perspective Zhengxiao Du Aohan Zeng Yuxiao Dong Jie Tang UQCV LRM 73 46 0 23 Mar 2024
The Garden of Forking Paths: Observing Dynamic Parameters Distribution in Large Language Models Carlo Nicolini Jacopo Staiano Bruno Lepri Raffaele Marino MoE 34 1 0 13 Mar 2024
Tune without Validation: Searching for Learning Rate and Weight Decay on Training Sets Lorenzo Brigato Stavroula Mougiakakou 45 0 0 08 Mar 2024
Uncovering Intermediate Variables in Transformers using Circuit Probing Michael A. Lepori Thomas Serre Ellie Pavlick 78 7 0 07 Nov 2023
Deep Neural Networks for Automatic Speaker Recognition Do Not Learn Supra-Segmental Temporal Features Daniel Neururer Volker Dellwo Thilo Stadelmann 41 2 0 01 Nov 2023
Grokking as Compression: A Nonlinear Complexity Perspective Ziming Liu Ziqian Zhong Max Tegmark 38 9 0 09 Oct 2023
Grokking as a First Order Phase Transition in Two Layer Networks Noa Rubin Inbar Seroussi Zohar Ringel 37 16 0 05 Oct 2023
Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data Zhiwei Xu Yutong Wang Spencer Frei Gal Vardi Wei Hu MLT 28 24 0 04 Oct 2023
DeepDecipher: Accessing and Investigating Neuron Activation in Large Language Models Albert Garde Esben Kran Fazl Barez 19 2 0 03 Oct 2023
Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck Benjamin L. Edelman Surbhi Goel Sham Kakade Eran Malach Cyril Zhang 48 8 0 07 Sep 2023
SALSA VERDE: a machine learning attack on Learning With Errors with sparse small secrets Cathy Li Emily Wenger Zeyuan Allen-Zhu François Charton Kristin E. Lauter AAML 33 10 0 20 Jun 2023
Adaptive Conformal Regression with Jackknife+ Rescaled Scores N. Deutschmann Mattia Rigotti María Rodríguez Martínez 26 10 0 31 May 2023
Faith and Fate: Limits of Transformers on Compositionality Nouha Dziri Ximing Lu Melanie Sclar Xiang Lorraine Li Liwei Jian ... Sean Welleck Xiang Ren Allyson Ettinger Zaïd Harchaoui Yejin Choi ReLM LRM 32 336 0 29 May 2023
Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models Yuhui Zhang Michihiro Yasunaga Zhengping Zhou Jeff Z. HaoChen James Zou Percy Liang Serena Yeung 47 7 0 27 May 2023
Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability Ziming Liu Eric Gan Max Tegmark 26 36 0 04 May 2023
Emergent and Predictable Memorization in Large Language Models Stella Biderman USVSN Sai Prashanth Lintang Sutawika Hailey Schoelkopf Quentin G. Anthony Shivanshu Purohit Edward Raf 35 117 0 21 Apr 2023