Title
Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models? Sigma Jahan Mohammad Masudur Rahman 20 0 0 09 Jun 2025
dots.llm1 Technical Report Bi Huo Bin Tu Cheng Qin Da Zheng Debing Zhang ... Yuqiu Ji Ze Wen Zhenhai Liu Zichao Li Zilong Liao MoE 53 0 0 06 Jun 2025
GradPower: Powering Gradients for Faster Language Model Pre-Training Mingze Wang Jinbo Wang Jiaqi Zhang Wei Wang Peng Pei Xunliang Cai Weinan E Lei Wu 58 0 0 30 May 2025
Learning in Compact Spaces with Approximately Normalized Transformers Jörg Franke Urs Spiegelhalter Marianna Nezhurina J. Jitsev Frank Hutter Michael Hefenbrock 60 0 0 28 May 2025
Don't Look Only Once: Towards Multimodal Interactive Reasoning with Selective Visual Revisitation Jiwan Chung Junhyeok Kim Siyeol Kim Jaeyoung Lee Min Soo Kim Youngjae Yu LRM 95 0 0 24 May 2025
Short-Range Dependency Effects on Transformer Instability and a Decomposed Attention Solution Suvadeep Hajra 103 0 0 21 May 2025
Scalable Autoregressive 3D Molecule Generation Austin H. Cheng Chong Sun Alán Aspuru-Guzik 100 1 0 20 May 2025
Scale-invariant Attention Ben Anson Xi Wang Laurence Aitchison LRM 105 0 0 20 May 2025
Learning by solving differential equations Benoit Dherin Michael Munn Hanna Mazzawi Michael Wunder Sourabh Medapati Javier Gonzalvo 60 0 0 19 May 2025
Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training Shane Bergsma Nolan Dey Gurpreet Gosal Gavia Gray Daria Soboleva Joel Hestness 80 2 0 19 May 2025
Don't be lazy: CompleteP enables compute-efficient deep transformers Nolan Dey Bin Claire Zhang Lorenzo Noci Mufan Li Blake Bordelon Shane Bergsma Cengiz Pehlevan Boris Hanin Joel Hestness 118 2 0 02 May 2025
Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model Zhiwei Xu Zhiyu Ni Yixin Wang Wei Hu CLL 105 2 0 17 Apr 2025
SEA-LION: Southeast Asian Languages in One Network Raymond Ng Thanh Ngan Nguyen Yuli Huang Ngee Chia Tai Wai Yi Leong ... David Ong Tat-Wee B. Liu William-Chandra Tjhi Min Zhang Leslie Teo 131 14 0 08 Apr 2025
Efficient Construction of Model Family through Progressive Training Using Model Expansion Kazuki Yano Sho Takase Sosuke Kobayashi Shun Kiyono Jun Suzuki 100 0 0 01 Apr 2025
Unified Multimodal Discrete Diffusion Alexander Swerdlow Mihir Prabhudesai Siddharth Gandhi Deepak Pathak Katerina Fragkiadaki DiffM 125 8 0 26 Mar 2025
Gemma 3 Technical Report Gemma Team Aishwarya B Kamath Johan Ferret Shreya Pathak Nino Vieillard ... Harshal Tushar Lehri Hussein Hazimeh Ian Ballantyne Idan Szpektor Ivan Nardini VLM 193 136 0 25 Mar 2025
Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo Zachary B. Charles Gabriel Teston Lucio Dery Keith Rush Nova Fallen Zachary Garrett Arthur Szlam Arthur Douillard 459 6 0 12 Mar 2025
DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in Multimodal Cycles Rui Zhao Weijia Mao Mike Zheng Shou 107 1 0 05 Mar 2025
Proteina: Scaling Flow-based Protein Structure Generative Models Tomas Geffner Kieran Didi Zuobai Zhang Danny Reidenbach Zhonglin Cao ... Mario Geiger Christian Dallago E. Küçükbenli Arash Vahdat Karsten Kreis DiffM AI4CE 89 12 0 02 Mar 2025
Kanana: Compute-efficient Bilingual Language Models Kanana LLM Team Yunju Bak Hojin Lee Minho Ryu Jiyeon Ham ... Daniel Lee Minchul Lee MinHyung Lee Shinbok Lee Gaeun Seo 177 1 0 26 Feb 2025
Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam Tianjin Huang Haotian Hu Zhenyu Zhang Gaojie Jin Xianrui Li ... Tianlong Chen Lu Liu Qingsong Wen Zhangyang Wang Shiwei Liu MQ 116 2 0 24 Feb 2025
Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs Shane Bergsma Nolan Dey Gurpreet Gosal Gavia Gray Daria Soboleva Joel Hestness 109 8 0 21 Feb 2025
PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflicts Zeman Li Yuan Deng Peilin Zhong Meisam Razaviyayn Vahab Mirrokni MoMe 147 1 0 10 Feb 2025
UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths Weijia Mao Zhiyong Yang Mike Zheng Shou MoE 198 1 0 10 Feb 2025
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models Samira Abnar Harshay Shah Dan Busbridge Alaaeldin Mohamed Elnouby Ali J. Susskind Vimal Thilak MoE LRM 154 10 0 21 Jan 2025
Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit Oleg Filatov Jan Ebert Jiangtao Wang Stefan Kesselheim 115 4 0 10 Jan 2025
Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism Tim Tsz-Kit Lau Weijian Li Chenwei Xu Han Liu Mladen Kolar 466 0 0 30 Dec 2024
Loss-to-Loss Prediction: Scaling Laws for All Datasets David Brandfonbrener Nikhil Anand Nikhil Vyas Eran Malach Sham Kakade 132 4 0 19 Nov 2024
Scaling Laws for Precision Tanishq Kumar Zachary Ankner Benjamin Spector Blake Bordelon Niklas Muennighoff Mansheej Paul Cengiz Pehlevan Christopher Ré Aditi Raghunathan AIFin MoMe 106 29 0 07 Nov 2024
Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers Gavia Gray Aman Tiwari Shane Bergsma Joel Hestness 94 2 0 01 Nov 2024
Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training Atli Kosson Bettina Messmer Martin Jaggi AI4CE 71 5 0 31 Oct 2024
How Does Critical Batch Size Scale in Pre-training? Hanlin Zhang Depen Morwani Nikhil Vyas Jingfeng Wu Difan Zou Udaya Ghai Dean Phillips Foster Sham Kakade 189 18 0 29 Oct 2024
Methods of improving LLM training stability Oleg Rybakov Mike Chrzanowski Peter Dykas Jinze Xue Ben Lanir 80 1 0 22 Oct 2024
A Hitchhiker's Guide to Scaling Law Estimation Leshem Choshen Yang Zhang Jacob Andreas 103 8 0 15 Oct 2024
Spatio-Temporal Control for Masked Motion Synthesis Ekkasit Pinyoanuntapong Muhammad Usama Saleem Korrawe Karunratanakul Pu Wang Hongfei Xue Chong Chen Chuan Guo Junli Cao J. Ren Sergey Tulyakov VGen 92 7 0 14 Oct 2024
Scaling Laws For Diffusion Transformers Zhengyang Liang Hao He Ceyuan Yang Bo Dai 89 14 0 10 Oct 2024
PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency Preferred Elements : Kenshin Abe Kaizaburo Chubachi Yasuhiro Fujita ... Yoshihiko Ozaki Shotaro Sano Shuji Suzuki Tianqi Xu Toshihiko Yanase 87 0 0 10 Oct 2024
Falcon Mamba: The First Competitive Attention-free 7B Language Model Jingwei Zuo Maksim Velikanov Dhia Eddine Rhaiem Ilyas Chahed Younes Belkada Guillaume Kunsch Hakim Hacid ALM 89 17 0 07 Oct 2024
DEPT: Decoupled Embeddings for Pre-training Language Models Alex Iacob Lorenzo Sani Meghdad Kurmanji William F. Shen Xinchi Qiu Dongqi Cai Yan Gao Nicholas D. Lane VLM 620 1 0 07 Oct 2024
Output Scouting: Auditing Large Language Models for Catastrophic Responses Andrew Bell Joao Fonseca KELM 145 2 0 04 Oct 2024
Scaling Optimal LR Across Token Horizons Johan Bjorck Alon Benhaim Vishrav Chaudhary Furu Wei Xia Song 148 8 0 30 Sep 2024
Hyper-Connections Defa Zhu Hongzhi Huang Zihao Huang Yutao Zeng Yunyao Mao Banggu Wu Qiyang Min Xun Zhou 87 6 0 29 Sep 2024
CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair Mingjie Liu Yun-Da Tsai Wenfei Zhou Haoxing Ren SyDa 3DV 118 17 0 19 Sep 2024
SOAP: Improving and Stabilizing Shampoo using Adam Nikhil Vyas Depen Morwani Rosie Zhao Itai Shapira David Brandfonbrener Lucas Janson Sham Kakade Sham Kakade 165 38 0 17 Sep 2024
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation Jinheng Xie Weijia Mao Zechen Bai David Junhao Zhang Weihao Wang Kevin Qinghong Lin Yuchao Gu Zhijie Chen Zhenheng Yang Mike Zheng Shou 138 228 0 22 Aug 2024
Apple Intelligence Foundation Language Models Tom Gunter Zirui Wang Chong-Jun Wang Ruoming Pang Andy Narayanan ... Xinwen Liu Yang Zhao Yin Xia Zhile Ren Zhongzheng Ren 141 40 0 29 Jul 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 123 10 0 24 Jul 2024
Deconstructing What Makes a Good Optimizer for Language Models Rosie Zhao Depen Morwani David Brandfonbrener Nikhil Vyas Sham Kakade 148 25 0 10 Jul 2024
Stepping on the Edge: Curvature Aware Learning Rate Tuners Vincent Roulet Atish Agarwala Jean-Bastien Grill Grzegorz Swirszcz Mathieu Blondel Fabian Pedregosa 99 3 0 08 Jul 2024
Normalization and effective learning rates in reinforcement learning Clare Lyle Zeyu Zheng Khimya Khetarpal James Martens H. V. Hasselt Razvan Pascanu Will Dabney 97 13 0 01 Jul 2024