Exploring Low Rank Training of Deep Neural Networks

Exploring Low Rank Training of Deep Neural Networks

27 September 2022

Siddhartha Rao Kamalakara

Acyr F. Locatelli

Bharat Venkitesh

Jimmy Ba

Papers citing "Exploring Low Rank Training of Deep Neural Networks"

5 / 5 papers shown

Title
Mixture of Parrots: Experts improve memorization more than reasoning Samy Jelassi Clara Mohri David Brandfonbrener Alex Gu Nikhil Vyas Nikhil Anand David Alvarez-Melis Yuanzhi Li Sham Kakade Eran Malach MoE 33 4 0 24 Oct 2024
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection Jiawei Zhao Zhenyu (Allen) Zhang Beidi Chen Zhangyang Wang A. Anandkumar Yuandong Tian 43 175 0 06 Mar 2024
Flat Minima in Linear Estimation and an Extended Gauss Markov Theorem Simon Segert 32 0 0 18 Nov 2023
Point Cloud Network: An Order of Magnitude Improvement in Linear Layer Parameter Count Charles Hetterich 3DPC 12 0 0 22 Sep 2023
Initialization and Regularization of Factorized Neural Layers M. Khodak Neil A. Tenenholtz Lester W. Mackey Nicolò Fusi 65 56 0 03 May 2021