ReLU's Revival: On the Entropic Overload in Normalization-Free Large
Language Models

v1v2v3 (latest)

ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models

12 October 2024

ArXiv (abs)PDF HTML Github

Papers citing "ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models"

14 / 14 papers shown

Title
HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization Zhijian Zhuo Yutao Zeng Ya Wang Sijun Zhang Jian Yang Xiaoqing Li Xun Zhou Jinwen Ma 79 0 0 06 Mar 2025
The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry Michael Zhang Kush S. Bhatia Hermann Kumbong Christopher Ré 66 54 0 06 Feb 2024
Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning Bingchen Zhao Haoqin Tu Chen Wei Jieru Mei Cihang Xie 82 36 0 18 Dec 2023
Linear Log-Normal Attention with Unbiased Concentration Yury Nahshan Dor-Joseph Kampeas E. Haleva 59 8 0 22 Nov 2023
Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption Itamar Zimerman Moran Baruch Nir Drucker Gilad Ezov Omri Soceanu Lior Wolf 80 17 0 15 Nov 2023
The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit Lorenzo Noci Chuning Li Mufan Li Bobby He Thomas Hofmann Chris J. Maddison Daniel M. Roy 90 36 0 30 Jun 2023
DeepReShape: Redesigning Neural Networks for Efficient Private Inference N. Jha Brandon Reagen 67 10 0 20 Apr 2023
Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation Bobby He James Martens Guodong Zhang Aleksandar Botev Andy Brock Samuel L. Smith Yee Whye Teh 79 30 0 20 Feb 2023
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models Xiuying Wei Yunchen Zhang Xiangguo Zhang Ruihao Gong Shanghang Zhang Qi Zhang F. Yu Xianglong Liu MQ 100 152 0 27 Sep 2022
Block-Recurrent Transformers DeLesley S. Hutchins Imanol Schlag Yuhuai Wu Ethan Dyer Behnam Neyshabur 87 100 0 11 Mar 2022
BERT Busters: Outlier Dimensions that Disrupt Transformers Olga Kovaleva Saurabh Kulshreshtha Anna Rogers Anna Rumshisky 93 92 0 14 May 2021
DeepReDuce: ReLU Reduction for Fast Private Inference N. Jha Zahra Ghodsi S. Garg Brandon Reagen 79 91 0 02 Mar 2021
On Layer Normalization in the Transformer Architecture Ruibin Xiong Yunchang Yang Di He Kai Zheng Shuxin Zheng Chen Xing Huishuai Zhang Yanyan Lan Liwei Wang Tie-Yan Liu AI4CE 142 996 0 12 Feb 2020
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 432 10,531 0 21 Jul 2016

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.