Disentangling the Causes of Plasticity Loss in Neural Networks

29 February 2024

Papers citing "Disentangling the Causes of Plasticity Loss in Neural Networks"

34 / 34 papers shown

Title
Understanding and Exploiting Plasticity for Non-stationary Network Resource Adaptation Zhiqiang He Zhi Liu CLL 96 0 0 02 May 2025
Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC Tyler Clark Mark Towers Christine Evers Jonathon Hare OffRL 138 1 0 06 Nov 2024
MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL C. Voelcker Marcel Hussing Eric Eaton Amir-massoud Farahmand Igor Gilitschenski 96 4 0 11 Oct 2024
Neuroplastic Expansion in Deep Reinforcement Learning Jiashun Liu J. Obando-Ceron Rameswar Panda L. Pan 89 6 0 10 Oct 2024
Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL Ghada Sokar J. Obando-Ceron Rameswar Panda Hugo Larochelle Pablo Samuel Castro MoE 310 7 0 02 Oct 2024
Can Learned Optimization Make Reinforcement Learning Less Difficult? Alexander David Goldie Chris Xiaoxuan Lu Matthew Jackson Shimon Whiteson Jakob N. Foerster 111 4 0 09 Jul 2024
Simplifying Deep Temporal Difference Learning Matteo Gallici Mattie Fellows Benjamin Ellis B. Pou Ivan Masmitja Jakob Foerster Mario Martin OffRL 133 26 0 05 Jul 2024
Small-scale proxies for large-scale Transformer training instabilities Mitchell Wortsman Peter J. Liu Lechao Xiao Katie Everett A. Alemi ... Jascha Narain Sohl-Dickstein Kelvin Xu Jaehoon Lee Justin Gilmer Simon Kornblith 87 99 0 25 Sep 2023
Continual Learning as Computationally Constrained Reinforcement Learning Saurabh Kumar Henrik Marklund Anand Srinivasa Rao Yifan Zhu Hong Jun Jeon Yueyang Liu Benjamin Van Roy CLL 65 24 0 10 Jul 2023
PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning Hojoon Lee Hanseul Cho Hyunseung Kim Daehoon Gwak Joonkee Kim Jaegul Choo Se-Young Yun Chulhee Yun OffRL 128 29 0 19 Jun 2023
Deep Reinforcement Learning with Plasticity Injection Evgenii Nikishin Junhyuk Oh Georg Ostrovski Clare Lyle Razvan Pascanu Will Dabney André Barreto OffRL 47 52 0 24 May 2023
Understanding plasticity in neural networks Clare Lyle Zeyu Zheng Evgenii Nikishin Bernardo Avila-Pires Razvan Pascanu Will Dabney AI4CE 105 104 0 02 Mar 2023
Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation Bobby He James Martens Guodong Zhang Aleksandar Botev Andy Brock Samuel L. Smith Yee Whye Teh 79 30 0 20 Feb 2023
Understanding and Preventing Capacity Loss in Reinforcement Learning Clare Lyle Mark Rowland Will Dabney CLL 84 113 0 20 Apr 2022
Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers Guodong Zhang Aleksandar Botev James Martens OffRL 73 28 0 15 Mar 2022
A study on the plasticity of neural networks Tudor Berariu Wojciech M. Czarnecki Soham De J. Bornschein Samuel L. Smith Razvan Pascanu Claudia Clopath CLL AI4CE 61 32 0 31 May 2021
Decoupling Value and Policy for Generalization in Reinforcement Learning Roberta Raileanu Rob Fergus DRL OffRL 65 99 0 20 Feb 2021
On the Origin of Implicit Regularization in Stochastic Gradient Descent Samuel L. Smith Benoit Dherin David Barrett Soham De MLT 47 204 0 28 Jan 2021
WILDS: A Benchmark of in-the-Wild Distribution Shifts Pang Wei Koh Shiori Sagawa Henrik Marklund Sang Michael Xie Marvin Zhang ... A. Kundaje Emma Pierson Sergey Levine Chelsea Finn Percy Liang OOD 225 1,445 0 14 Dec 2020
Gradient Starvation: A Learning Proclivity in Neural Networks Mohammad Pezeshki Sekouba Kaba Yoshua Bengio Aaron Courville Doina Precup Guillaume Lajoie MLT 131 268 0 18 Nov 2020
Implicit Gradient Regularization David Barrett Benoit Dherin 76 152 0 23 Sep 2020
Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning Maximilian Igl Gregory Farquhar Jelena Luketina Wendelin Boehmer Shimon Whiteson 81 88 0 10 Jun 2020
Understanding and Improving Layer Normalization Jingjing Xu Xu Sun Zhiyuan Zhang Guangxiang Zhao Junyang Lin FAtt 95 356 0 16 Nov 2019
On the Impact of the Activation Function on Deep Neural Networks Training Soufiane Hayou Arnaud Doucet Judith Rousseau ODL 65 200 0 19 Feb 2019
Shampoo: Preconditioned Stochastic Tensor Optimization Vineet Gupta Tomer Koren Y. Singer ODL 92 226 0 26 Feb 2018
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor Tuomas Haarnoja Aurick Zhou Pieter Abbeel Sergey Levine 317 8,406 0 04 Jan 2018
Deep Neural Networks as Gaussian Processes Jaehoon Lee Yasaman Bahri Roman Novak S. Schoenholz Jeffrey Pennington Jascha Narain Sohl-Dickstein UQCV BDL 135 1,099 0 01 Nov 2017
A Distributional Perspective on Reinforcement Learning Marc G. Bellemare Will Dabney Rémi Munos OffRL 101 1,506 0 21 Jul 2017
The Shattered Gradients Problem: If resnets are the answer, then what is the question? David Balduzzi Marcus Frean Lennox Leary J. P. Lewis Kurt Wan-Duo Ma Brian McWilliams ODL 73 406 0 28 Feb 2017
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 426 10,531 0 21 Jul 2016
Exponential expressivity in deep neural networks through transient chaos Ben Poole Subhaneil Lahiri M. Raghu Jascha Narain Sohl-Dickstein Surya Ganguli 90 595 0 16 Jun 2016
Learning values across many orders of magnitude H. V. Hasselt A. Guez Matteo Hessel Volodymyr Mnih David Silver 65 170 0 24 Feb 2016
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe Christian Szegedy OOD 465 43,341 0 11 Feb 2015
On the Number of Linear Regions of Deep Neural Networks Guido Montúfar Razvan Pascanu Kyunghyun Cho Yoshua Bengio 92 1,256 0 08 Feb 2014