On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization

19 February 2018

Papers citing "On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization"

50 / 119 papers shown

Title
Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks Noam Razin Asaf Maman Nadav Cohen 46 29 0 27 Jan 2022
DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization Aviral Kumar Rishabh Agarwal Tengyu Ma Aaron Courville George Tucker Sergey Levine OffRL 31 65 0 09 Dec 2021
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models Tri Dao Beidi Chen Kaizhao Liang Jiaming Yang Zhao Song Atri Rudra Christopher Ré 33 75 0 30 Nov 2021
Neural Networks as Kernel Learners: The Silent Alignment Effect Alexander B. Atanasov Blake Bordelon Cengiz Pehlevan MLT 26 75 0 29 Oct 2021
Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networks Tolga Ergen Mert Pilanci 32 16 0 18 Oct 2021
S-Cyc: A Learning Rate Schedule for Iterative Pruning of ReLU-based Networks Shiyu Liu Chong Min John Tan Mehul Motani CLL 29 4 0 17 Oct 2021
Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programs Tolga Ergen Mert Pilanci OffRL MLT 32 33 0 11 Oct 2021
Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations Jiayao Zhang Hua Wang Weijie J. Su 35 7 0 11 Oct 2021
Towards Demystifying Representation Learning with Non-contrastive Self-supervision Xiang Wang Xinlei Chen S. Du Yuandong Tian SSL 21 26 0 11 Oct 2021
Multi-Head ReLU Implicit Neural Representation Networks Arya Aftab Alireza Morsali 42 11 0 07 Oct 2021
EF21 with Bells & Whistles: Practical Algorithmic Extensions of Modern Error Feedback Ilyas Fatkhullin Igor Sokolov Eduard A. Gorbunov Zhize Li Peter Richtárik 46 46 0 07 Oct 2021
Stochastic Training is Not Necessary for Generalization Jonas Geiping Micah Goldblum Phillip E. Pope Michael Moeller Tom Goldstein 89 72 0 29 Sep 2021
TrAISformer -- A Transformer Network with Sparse Augmented Data Representation and Cross Entropy Loss for AIS-based Vessel Trajectory Prediction Duong Nguyen Ronan Fablet 21 24 0 08 Sep 2021
Identifiable Energy-based Representations: An Application to Estimating Heterogeneous Causal Effects Yao Zhang Jeroen Berrevoets M. Schaar CML 36 5 0 06 Aug 2021
A Theoretical Analysis of Fine-tuning with Linear Teachers Gal Shachaf Alon Brutzkus Amir Globerson 34 17 0 04 Jul 2021
A Mechanism for Producing Aligned Latent Spaces with Autoencoders Saachi Jain Adityanarayanan Radhakrishnan Caroline Uhler 24 9 0 29 Jun 2021
Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction Dominik Stöger Mahdi Soltanolkotabi ODL 42 75 0 28 Jun 2021
Layer Folding: Neural Network Depth Reduction using Activation Linearization Amir Ben Dror Niv Zehngut Avraham Raviv E. Artyomov Ran Vitek R. Jevnisek 29 20 0 17 Jun 2021
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers Rabeeh Karimi Mahabadi James Henderson Sebastian Ruder MoE 67 468 0 08 Jun 2021
Initialization and Regularization of Factorized Neural Layers M. Khodak Neil A. Tenenholtz Lester W. Mackey Nicolò Fusi 65 56 0 03 May 2021
PCFGs Can Do Better: Inducing Probabilistic Context-Free Grammars with Many Symbols Aaron Courville Yanpeng Zhao Kewei Tu 23 22 0 28 Apr 2021
Understanding self-supervised Learning Dynamics without Contrastive Pairs Yuandong Tian Xinlei Chen Surya Ganguli SSL 138 281 0 12 Feb 2021
A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks Asaf Noy Yi Tian Xu Y. Aflalo Lihi Zelnik-Manor Rong Jin 39 3 0 12 Jan 2021
Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks Xiangyu Chang Yingcong Li Samet Oymak Christos Thrampoulidis 35 50 0 16 Dec 2020
Neural collapse with unconstrained features D. Mixon Hans Parshall Jianzong Pi 28 114 0 23 Nov 2020
Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting Zeke Xie Fengxiang He Shaopeng Fu Issei Sato Dacheng Tao Masashi Sugiyama 21 60 0 12 Nov 2020
Modular Primitives for High-Performance Differentiable Rendering S. Laine Janne Hellsten Tero Karras Yeongho Seol J. Lehtinen Timo Aila 6 436 0 06 Nov 2020
Deep matrix factorizations Pierre De Handschutter Nicolas Gillis Xavier Siebert BDL 28 40 0 01 Oct 2020
It's Hard for Neural Networks To Learn the Game of Life Jacob Mitchell Springer Garrett Kenyon 21 21 0 03 Sep 2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding Dmitry Lepikhin HyoukJoong Lee Yuanzhong Xu Dehao Chen Orhan Firat Yanping Huang M. Krikun Noam M. Shazeer Z. Chen MoE 43 1,108 0 30 Jun 2020
Recurrent Quantum Neural Networks Johannes Bausch 21 151 0 25 Jun 2020
Auxiliary Learning by Implicit Differentiation Aviv Navon Idan Achituve Haggai Maron Gal Chechik Ethan Fetaya 28 59 0 22 Jun 2020
DO-Conv: Depthwise Over-parameterized Convolutional Layer Jinming Cao Yangyan Li Mingchao Sun Ying-Cong Chen Dani Lischinski Daniel Cohen-Or Baoquan Chen Changhe Tu OOD 33 166 0 22 Jun 2020
Exploring Weight Importance and Hessian Bias in Model Pruning Mingchen Li Yahya Sattar Christos Thrampoulidis Samet Oymak 28 3 0 19 Jun 2020
Directional Pruning of Deep Neural Networks Shih-Kang Chao Zhanyu Wang Yue Xing Guang Cheng ODL 21 33 0 16 Jun 2020
Knowledge Distillation: A Survey Jianping Gou B. Yu Stephen J. Maybank Dacheng Tao VLM 19 2,843 0 09 Jun 2020
Gradient Monitored Reinforcement Learning Mohammed Sharafath Abdul Hameed Gavneet Singh Chadha Andreas Schwung S. Ding 33 10 0 25 May 2020
Implicit Regularization in Deep Learning May Not Be Explainable by Norms Noam Razin Nadav Cohen 24 155 0 13 May 2020
The Landscape of Matrix Factorization Revisited Hossein Valavi Sulin Liu Peter J. Ramadge 17 5 0 27 Feb 2020
Convex Geometry and Duality of Over-parameterized Neural Networks Tolga Ergen Mert Pilanci MLT 42 54 0 25 Feb 2020
Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinity Shiyu Liang Ruoyu Sun R. Srikant 35 19 0 31 Dec 2019
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms Kaipeng Zhang Zhuoran Yang Tamer Basar 58 1,181 0 24 Nov 2019
Neural Similarity Learning Weiyang Liu Zhen Liu James M. Rehg Le Song 27 29 0 28 Oct 2019
Overparameterized Neural Networks Implement Associative Memory Adityanarayanan Radhakrishnan M. Belkin Caroline Uhler BDL 35 71 0 26 Sep 2019
Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization Xinyan Li Qilong Gu Yingxue Zhou Tiancong Chen A. Banerjee ODL 42 51 0 24 Jul 2019
Associated Learning: Decomposing End-to-end Backpropagation based on Auto-encoders and Target Propagation Yu-Wei Kao Hung-Hsuan Chen BDL 17 5 0 13 Jun 2019
Implicit Regularization in Deep Matrix Factorization Sanjeev Arora Nadav Cohen Wei Hu Yuping Luo AI4CE 38 491 0 31 May 2019
On the Expressive Power of Deep Polynomial Neural Networks Joe Kileel Matthew Trager Joan Bruna 24 82 0 29 May 2019
Every Local Minimum Value is the Global Minimum Value of Induced Model in Non-convex Machine Learning Kenji Kawaguchi Jiaoyang Huang L. Kaelbling AAML 24 18 0 07 Apr 2019
Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks Mingchen Li Mahdi Soltanolkotabi Samet Oymak NoLa 47 351 0 27 Mar 2019