All you need is a good init

19 November 2015

Papers citing "All you need is a good init"

50 / 81 papers shown

Title
Shrinkage Initialization for Smooth Learning of Neural Networks Miao Cheng Feiyan Zhou Hongwei Zou Limin Wang AI4CE 31 0 0 12 Apr 2025
RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation Jun Liu Zhenglun Kong Peiyan Dong Changdi Yang Xuan Shen ... Wei Niu Wenbin Zhang Xue Lin Dong Huang Yanzhi Wang ALM 36 2 0 08 Jan 2025
Residual Kolmogorov-Arnold Network for Enhanced Deep Learning Ray Congrui Yu Sherry Wu Jiang Gui 38 1 0 07 Oct 2024
Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis Hyunwoo Lee Hayoung Choi Hyunju Kim 39 1 0 03 Oct 2024
From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks Clémentine Dominé Nicolas Anguita A. Proca Lukas Braun D. Kunin P. Mediano Andrew M. Saxe 30 3 0 22 Sep 2024
Oja's plasticity rule overcomes several challenges of training neural networks under biological constraints Navid Shervani-Tabar Marzieh Alireza Mirhoseini Robert Rosenbaum AAML AI4CE 37 0 0 15 Aug 2024
Advancing Neural Network Performance through Emergence-Promoting Initialization Scheme Johnny Jingze Li V. George Gabriel A. Silva ODL 39 0 0 26 Jul 2024
Initializing Models with Larger Ones Zhiqiu Xu Yanjie Chen Kirill Vishniakov Yida Yin Zhiqiang Shen Trevor Darrell Lingjie Liu Zhuang Liu 30 17 0 30 Nov 2023
Maestro: Uncovering Low-Rank Structures via Trainable Decomposition Samuel Horváth Stefanos Laskaridis Shashank Rajput Hongyi Wang BDL 32 4 0 28 Aug 2023
Fading memory as inductive bias in residual recurrent networks I. Dubinin Felix Effenberger 38 4 0 27 Jul 2023
A Stable and Scalable Method for Solving Initial Value PDEs with Neural Networks Marc Finzi Andres Potapczynski M. Choptuik A. Wilson 13 12 0 28 Apr 2023
Best Practices for 2-Body Pose Forecasting Muhammad Rameez Ur Rahman Luca Scofano Edoardo De Matteis Alessandro Flaborea Alessio Sampieri Fabio Galasso 26 9 0 12 Apr 2023
Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning Nikhil Singh Chih-Wei Wu Iroro Orife Mahdi M. Kalayeh 23 2 0 12 Apr 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 28 40 0 07 Apr 2023
Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruning Huan Wang Can Qin Yue Bai Yun Fu 32 20 0 12 Jan 2023
Orthogonal SVD Covariance Conditioning and Latent Disentanglement Yue Song N. Sebe Wei Wang 26 6 0 11 Dec 2022
Characterizing the Spectrum of the NTK via a Power Series Expansion Michael Murray Hui Jin Benjamin Bowman Guido Montúfar 30 11 0 15 Nov 2022
Dynamical Isometry for Residual Networks Advait Gadhikar R. Burkholz ODL AI4CE 32 2 0 05 Oct 2022
Batch Normalization Explained Randall Balestriero Richard G. Baraniuk AAML 28 16 0 29 Sep 2022
Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality Yue Song N. Sebe Wei Wang 13 8 0 05 Jul 2022
Entangled Residual Mappings Mathias Lechner Ramin Hasani Z. Babaiee Radu Grosu Daniela Rus T. Henzinger Sepp Hochreiter 6 4 0 02 Jun 2022
Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs Fanchen Bu D. Chang 17 6 0 12 May 2022
Enhancing Adversarial Training with Second-Order Statistics of Weights Gao Jin Xinping Yi Wei Huang S. Schewe Xiaowei Huang AAML 17 47 0 11 Mar 2022
Random vector functional link network: recent developments, applications, and future directions Anil Kumar Malik Ruobin Gao M. A. Ganaie M. Tanveer Ponnuthurai Nagaratnam Suganthan 14 102 0 13 Feb 2022
Learning Enhancement of CNNs via Separation Index Maximizing at the First Convolutional Layer Ali Karimi Ahmad Kalhor SSL 11 0 0 13 Jan 2022
Ridgeless Interpolation with Shallow ReLU Networks in $1D$ is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functions Boris Hanin MLT 30 9 0 27 Sep 2021
The Unreasonable Effectiveness of the Final Batch Normalization Layer Veysel Kocaman O. M. Shir T. Baeck 13 1 0 18 Sep 2021
AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks G. Bingham Risto Miikkulainen ODL 24 4 0 18 Sep 2021
Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifiers Yunhui Guo Xudong Wang Yubei Chen Stella X. Yu 18 45 0 23 Jul 2021
Data-driven Weight Initialization with Sylvester Solvers Debasmit Das Yash Bhalgat Fatih Porikli ODL 22 3 0 02 May 2021
GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training Chen Zhu Renkun Ni Zheng Xu Kezhi Kong W. R. Huang Tom Goldstein ODL 31 53 0 16 Feb 2021
Advances in Electron Microscopy with Deep Learning Jeffrey M. Ede 29 2 0 04 Jan 2021
On Initial Pools for Deep Active Learning Akshay L Chandra Sai Vikas Desai Chaitanya Devaguptapu V. Balasubramanian 19 19 0 30 Nov 2020
A Study on the Uncertainty of Convolutional Layers in Deep Neural Networks Hao Shen Sihong Chen Ran Wang 22 5 0 27 Nov 2020
Nanopore Base Calling on the Edge Peter Perešíni V. Boža Broňa Brejová T. Vinař 11 38 0 09 Nov 2020
BYOL works even without batch statistics Pierre Harvey Richemond Jean-Bastien Grill Florent Altché Corentin Tallec Florian Strub ... Samuel L. Smith Soham De Razvan Pascanu Bilal Piot Michal Valko SSL 250 114 0 20 Oct 2020
SWIPENET: Object detection in noisy underwater images Long Chen Feixiang Zhou Shengke Wang Junyu Dong Ning Li Haiping Ma Xin Wang Huiyu Zhou 13 17 0 19 Oct 2020
Review: Deep Learning in Electron Microscopy Jeffrey M. Ede 26 79 0 17 Sep 2020
Persistent Neurons Yimeng Min 19 0 0 02 Jul 2020
Deep Isometric Learning for Visual Recognition Haozhi Qi Chong You X. Wang Yi-An Ma Jitendra Malik VLM 22 53 0 30 Jun 2020
New Interpretations of Normalization Methods in Deep Learning Jiacheng Sun Xiangyong Cao Hanwen Liang Weiran Huang Zewei Chen Zhenguo Li 11 34 0 16 Jun 2020
A Survey on Activation Functions and their relation with Xavier and He Normal Initialization Leonid Datta AI4CE 15 68 0 18 Mar 2020
A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks Zhaodong Chen Lei Deng Bangyan Wang Guoqi Li Yuan Xie 27 28 0 01 Jan 2020
Streaming convolutional neural networks for end-to-end learning with multi-megapixel images H. Pinckaers Bram van Ginneken G. Litjens MedIm 13 93 0 11 Nov 2019
Greedy Shallow Networks: An Approach for Constructing and Training Neural Networks Anton Dereventsov Armenak Petrosyan Clayton Webster 13 9 0 24 May 2019
On Graph Classification Networks, Datasets and Baselines Enxhell Luzhnica Ben Day Pietro Lió GNN 18 19 0 12 May 2019
On the security relevance of weights in deep learning Kathrin Grosse T. A. Trost Marius Mosbach Michael Backes Dietrich Klakow AAML 30 6 0 08 Feb 2019
Parameter Re-Initialization through Cyclical Batch Size Schedules Norman Mu Z. Yao A. Gholami Kurt Keutzer Michael W. Mahoney ODL 22 8 0 04 Dec 2018
Self-Referenced Deep Learning Xu Lan Xiatian Zhu S. Gong 19 23 0 19 Nov 2018
Good Initializations of Variational Bayes for Deep Models Simone Rossi Pietro Michiardi Maurizio Filippone BDL 17 21 0 18 Oct 2018