Scaling Laws Beyond Backpropagation

26 October 2022

Matthew J. Filipovich

Papers citing "Scaling Laws Beyond Backpropagation"

21 / 21 papers shown

Title
What Language Model to Train if You Have One Million GPU Hours? Teven Le Scao Thomas Wang Daniel Hesslow Lucile Saulnier Stas Bekman ... Lintang Sutawika Jaesung Tae Zheng-Xin Yong Julien Launay Iz Beltagy MoE AI4CE 261 107 0 27 Oct 2022
Revisiting Neural Scaling Laws in Language and Vision Ibrahim Alabdulmohsin Behnam Neyshabur Xiaohua Zhai 192 109 0 13 Sep 2022
Beyond neural scaling laws: beating power law scaling via data pruning Ben Sorscher Robert Geirhos Shashank Shekhar Surya Ganguli Ari S. Morcos 85 439 0 29 Jun 2022
Self-Supervised Pretraining for Differentially Private Learning Arash Asadian Evan Weidner Lei Jiang PICV 47 3 0 14 Jun 2022
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? Thomas Wang Adam Roberts Daniel Hesslow Teven Le Scao Hyung Won Chung Iz Beltagy Julien Launay Colin Raffel 83 173 0 12 Apr 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 441 6,222 0 05 Apr 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 185 1,944 0 29 Mar 2022
Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment M. Filipovich Zhimu Guo M. Al-Qadasi Bicky A. Marquez Hugh Morison V. Sorger Paul R. Prucnal S. Shekhar B. Shastri 43 54 0 12 Nov 2021
Scaling Laws for Neural Machine Translation Behrooz Ghorbani Orhan Firat Markus Freitag Ankur Bapna M. Krikun Xavier Garcia Ciprian Chelba Colin Cherry 63 101 0 16 Sep 2021
Do Transformer Modifications Transfer Across Implementations and Applications? Sharan Narang Hyung Won Chung Yi Tay W. Fedus Thibault Févry ... Wei Li Nan Ding Jake Marcus Adam Roberts Colin Raffel 72 127 0 23 Feb 2021
Adversarial Robustness by Design through Analog Computing and Synthetic Gradients Alessandro Cappelli Ruben Ohana Julien Launay Laurent Meunier Iacopo Poli Florent Krzakala AAML 109 13 0 06 Jan 2021
Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment Julien Launay Iacopo Poli Kilian Muller Gustave Pariente I. Carron L. Daudet Florent Krzakala S. Gigan MoE 43 18 0 11 Dec 2020
Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures Julien Launay Iacopo Poli Franccois Boniface Florent Krzakala 68 64 0 23 Jun 2020
Light-in-the-loop: using a photonics co-processor for scalable training of neural networks Julien Launay Iacopo Poli Kilian Muller I. Carron L. Daudet Florent Krzakala S. Gigan 39 6 0 02 Jun 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 555 4,797 0 23 Jan 2020
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data Guillaume Wenzek Marie-Anne Lachaux Alexis Conneau Vishrav Chaudhary Francisco Guzmán Armand Joulin Edouard Grave 81 654 0 01 Nov 2019
Principled Training of Neural Networks with Direct Feedback Alignment Julien Launay Iacopo Poli Florent Krzakala 43 35 0 11 Jun 2019
Training Neural Networks with Local Error Signals Arild Nøkland L. Eidnes 73 227 0 20 Jan 2019
Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures Sergey Bartunov Adam Santoro Blake A. Richards Luke Marris Geoffrey E. Hinton Timothy Lillicrap 82 244 0 12 Jul 2018
Deep Learning Scaling is Predictable, Empirically Joel Hestness Sharan Narang Newsha Ardalani G. Diamos Heewoo Jun Hassan Kianinejad Md. Mostofa Ali Patwary Yang Yang Yanqi Zhou 87 739 0 01 Dec 2017
Decoupled Neural Interfaces using Synthetic Gradients Max Jaderberg Wojciech M. Czarnecki Simon Osindero Oriol Vinyals Alex Graves David Silver Koray Kavukcuoglu 75 356 0 18 Aug 2016