A Trace-restricted Kronecker-Factored Approximation to Natural Gradient

21 November 2020

Papers citing "A Trace-restricted Kronecker-Factored Approximation to Natural Gradient"

8 / 8 papers shown

Title
COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs Liming Liu Zhenghao Xu Zixuan Zhang Hao Kang Zichong Li Chen Liang Weizhu Chen T. Zhao 117 1 0 24 Feb 2025
SOAP: Improving and Stabilizing Shampoo using Adam Nikhil Vyas Depen Morwani Rosie Zhao Itai Shapira David Brandfonbrener Lucas Janson Sham Kakade Sham Kakade 66 23 0 17 Sep 2024
A New Perspective on Shampoo's Preconditioner Depen Morwani Itai Shapira Nikhil Vyas Eran Malach Sham Kakade Lucas Janson 27 7 0 25 Jun 2024
On the Parameterization of Second-Order Optimization Effective Towards the Infinite Width Satoki Ishikawa Ryo Karakida 24 2 0 19 Dec 2023
Analysis and Comparison of Two-Level KFAC Methods for Training Deep Neural Networks Abdoulaye Koroko A. Anciaux-Sedrakian I. B. Gharbia Valérie Garès M. Haddou Quang-Huy Tran 17 0 0 31 Mar 2023
DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning Hyounguk Shon Janghyeon Lee Seungwook Kim Junmo Kim CLL 19 11 0 17 Aug 2022
Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning Lin Zhang S. Shi Wei Wang Bo-wen Li 28 10 0 30 Jun 2022
Second-Order Neural ODE Optimizer Guan-Horng Liu T. Chen Evangelos A. Theodorou 21 12 0 29 Sep 2021