AlphaDecay:Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs

AlphaDecay:Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs

17 June 2025

ArXiv (abs)PDF HTML

Papers citing "AlphaDecay:Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs"

11 / 11 papers shown

Title
Make Your LLM Fully Utilize the Context Shengnan An Zexiong Ma Zeqi Lin Nanning Zheng Jian-Guang Lou SyDa 95 66 0 25 Apr 2024
ReLoRA: High-Rank Training Through Low-Rank Updates Vladislav Lialin Namrata Shivagunde Sherin Muckatira Anna Rumshisky BDL 79 115 0 11 Jul 2023
A ConvNet for the 2020s Zhuang Liu Hanzi Mao Chaozheng Wu Christoph Feichtenhofer Trevor Darrell Saining Xie ViT 171 5,192 0 10 Jan 2022
Leveraging redundancy in attention with Reuse Transformers Srinadh Bhojanapalli Ayan Chakrabarti Andreas Veit Michal Lukasik Himanshu Jain Frederick Liu Yin-Wen Chang Sanjiv Kumar 43 26 0 13 Oct 2021
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients Juntang Zhuang Tommy M. Tang Yifan Ding S. Tatikonda Nicha Dvornek X. Papademetris James S. Duncan ODL 165 517 0 15 Oct 2020
Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data Charles H. Martin Tongsu Peng Peng Michael W. Mahoney 83 110 0 17 Feb 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 445 20,298 0 23 Oct 2019
Traditional and Heavy-Tailed Self Regularization in Neural Network Models Charles H. Martin Michael W. Mahoney 79 125 0 24 Jan 2019
Densely Connected Convolutional Networks Gao Huang Zhuang Liu Laurens van der Maaten Kilian Q. Weinberger PINN 3DV 775 36,861 0 25 Aug 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 194,322 0 10 Dec 2015
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan Andrew Zisserman FAtt MDE 1.7K 100,479 0 04 Sep 2014