Nonlinear Initialization Methods for Low-Rank Neural Networks

v1v2v3 (latest)

Nonlinear Initialization Methods for Low-Rank Neural Networks

2 February 2022

Kiran Vodrahalli

Rakesh Shivanna

M. Sathiamoorthy

ArXiv (abs)PDF HTML

Papers citing "Nonlinear Initialization Methods for Low-Rank Neural Networks"

15 / 15 papers shown

Title
Language model compression with weighted low-rank factorization Yen-Chang Hsu Ting Hua Sung-En Chang Qiang Lou Yilin Shen Hongxia Jin 66 108 0 30 Jun 2022
Scaling Law for Recommendation Models: Towards General-purpose User Representations Kyuyong Shin Hanock Kwak KyungHyun Kim Max Nihlén Ramström Jisu Jeong Jung-Woo Ha Seon Gyeom Kim ELM 105 42 0 15 Nov 2021
A Universal Law of Robustness via Isoperimetry Sébastien Bubeck Mark Sellke 55 218 0 26 May 2021
BASE Layers: Simplifying Training of Large, Sparse Models M. Lewis Shruti Bhosale Tim Dettmers Naman Goyal Luke Zettlemoyer MoE 208 283 0 30 Mar 2021
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps Tri Dao N. Sohoni Albert Gu Matthew Eichhorn Amit Blonder Megan Leszczynski Atri Rudra Christopher Ré 90 49 0 29 Dec 2020
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 188 1,604 0 30 Sep 2020
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors Michael W. Dusenberry Ghassen Jerfel Yeming Wen Yi-An Ma Jasper Snoek Katherine A. Heller Balaji Lakshminarayanan Dustin Tran UQCV BDL 79 215 0 14 May 2020
A Neural Scaling Law from the Dimension of the Data Manifold Utkarsh Sharma Jared Kaplan 79 53 0 22 Apr 2020
What is the State of Neural Network Pruning? Davis W. Blalock Jose Javier Gonzalez Ortiz Jonathan Frankle John Guttag 286 1,055 0 06 Mar 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 653 4,925 0 23 Jan 2020
DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures Huanrui Yang W. Wen H. Li 79 98 0 27 Aug 2019
Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon Xin Luna Dong Shangyu Chen Sinno Jialin Pan 191 507 0 22 May 2017
Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding Song Han Huizi Mao W. Dally 3DGS 263 8,864 0 01 Oct 2015
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification Kaiming He Xinming Zhang Shaoqing Ren Jian Sun VLM 358 18,661 0 06 Feb 2015
FitNets: Hints for Thin Deep Nets Adriana Romero Nicolas Ballas Samira Ebrahimi Kahou Antoine Chassang C. Gatta Yoshua Bengio FedML 332 3,906 0 19 Dec 2014