A Study of Gradient Variance in Deep Learning

A Study of Gradient Variance in Deep Learning

9 July 2020

David J. Fleet

Jimmy Ba

Papers citing "A Study of Gradient Variance in Deep Learning"

11 / 11 papers shown

Title
FedDuA: Doubly Adaptive Federated Learning Shokichi Takakura Seng Pei Liew Satoshi Hasegawa FedML 30 0 0 16 May 2025
Multiple Importance Sampling for Stochastic Gradient Estimation Corentin Salaün Xingchang Huang Iliyan Georgiev Niloy J. Mitra Gurprit Singh 32 1 0 22 Jul 2024
Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning E. Chimoto Jay Gala Orevaoghene Ahia Julia Kreutzer Bruce A. Bassett Sara Hooker VLM 55 4 0 29 May 2024
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics Shoaib Ahmed Siddiqui Nitarshan Rajkumar Tegan Maharaj David M. Krueger Sara Hooker 72 27 0 20 Sep 2022
On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity Vincent Szolnoky Viktor Andersson Balázs Kulcsár Rebecka Jörnsten 45 5 0 25 May 2022
MSTGD:A Memory Stochastic sTratified Gradient Descent Method with an Exponential Convergence Rate Aixiang Chen Chen Jinting Zhang Zanbo Zhang Zhihong Li 48 0 0 21 Feb 2022
On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications Ziqiao Wang Yongyi Mao FedML MLT 44 22 0 07 Oct 2021
Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization Alexandre Ramé Corentin Dancette Matthieu Cord OOD 56 206 0 07 Sep 2021
Estimating Example Difficulty Using Variance of Gradients Chirag Agarwal Daniel D'souza Sara Hooker 218 109 0 26 Aug 2020
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 312 2,900 0 15 Sep 2016
Efficient Per-Example Gradient Computations Ian Goodfellow 186 75 0 07 Oct 2015