Learned Optimizers that Scale and Generalize

14 March 2017

Olga Wichrowska

Niru Maheswaranathan

Matthew W. Hoffman

Sergio Gomez Colmenarejo

Misha Denil

Nando de Freitas

Jascha Narain Sohl-Dickstein

AI4CE

ArXiv PDF HTML

Papers citing "Learned Optimizers that Scale and Generalize"

50 / 79 papers shown

Title
QuickSplat: Fast 3D Surface Reconstruction via Learned Gaussian Initialization Yueh-Cheng Liu Lukas Höllein Matthias Nießner Angela Dai 3DGS 34 0 0 08 May 2025
Scalable Meta-Learning via Mixed-Mode Differentiation Iurii Kemaev Dan A Calian Luisa M Zintgraf Gregory Farquhar H. V. Hasselt 57 0 0 01 May 2025
Learning Versatile Optimizers on a Compute Diet A. Moudgil Boris Knyazev Guillaume Lajoie Eugene Belilovsky 214 0 0 22 Jan 2025
Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning Richa Upadhyay Ronald Phlypo Rajkumar Saini Marcus Liwicki 42 0 0 21 Jan 2025
Can Learned Optimization Make Reinforcement Learning Less Difficult? Alexander David Goldie Chris Xiaoxuan Lu Matthew Jackson Shimon Whiteson Jakob N. Foerster 44 3 0 09 Jul 2024
From Learning to Optimize to Learning Optimization Algorithms Camille Castera Peter Ochs 65 1 0 28 May 2024
MADA: Meta-Adaptive Optimizers through hyper-gradient Descent Kaan Ozkara Can Karakus Parameswaran Raman Mingyi Hong Shoham Sabach B. Kveton V. Cevher 30 2 0 17 Jan 2024
Advances and Challenges in Meta-Learning: A Technical Review Anna Vettoruzzo Mohamed-Rafik Bouguelia Joaquin Vanschoren Thorsteinn Rögnvaldsson K. Santosh OffRL 34 70 0 10 Jul 2023
B2Opt: Learning to Optimize Black-box Optimization with Little Budget Xiaobin Li K. Wu Xiaoyu Zhang Handing Wang Qingbin Liu 32 9 0 24 Apr 2023
Improving physics-informed neural networks with meta-learned optimization Alexander Bihlo PINN 36 18 0 13 Mar 2023
Symbolic Discovery of Optimization Algorithms Xiangning Chen Chen Liang Da Huang Esteban Real Kaiyuan Wang ... Xuanyi Dong Thang Luong Cho-Jui Hsieh Yifeng Lu Quoc V. Le 67 353 0 13 Feb 2023
Learning to Optimize for Reinforcement Learning Qingfeng Lan Rupam Mahmood Shuicheng Yan Zhongwen Xu OffRL 36 6 0 03 Feb 2023
Mnemosyne: Learning to Train Transformers with Transformers Deepali Jain K. Choromanski Kumar Avinava Dubey Sumeet Singh Vikas Sindhwani Tingnan Zhang Jie Tan OffRL 39 9 0 02 Feb 2023
A Nonstochastic Control Approach to Optimization Xinyi Chen Elad Hazan 47 5 0 19 Jan 2023
Federated Automatic Differentiation Keith Rush Zachary B. Charles Zachary Garrett FedML 34 1 0 18 Jan 2023
Learning-Assisted Algorithm Unrolling for Online Optimization with Budget Constraints Jianyi Yang Shaolei Ren 20 2 0 03 Dec 2022
Transformer-Based Learned Optimization Erik Gartner Luke Metz Mykhaylo Andriluka C. Freeman C. Sminchisescu 25 11 0 02 Dec 2022
Learning to Optimize with Dynamic Mode Decomposition Petr Simánek Daniel Vasata Pavel Kordík 31 5 0 29 Nov 2022
VeLO: Training Versatile Learned Optimizers by Scaling Up Luke Metz James Harrison C. Freeman Amil Merchant Lucas Beyer ... Naman Agrawal Ben Poole Igor Mordatch Adam Roberts Jascha Narain Sohl-Dickstein 35 60 0 17 Nov 2022
Learning to Learn with Generative Models of Neural Network Checkpoints William S. Peebles Ilija Radosavovic Tim Brooks Alexei A. Efros Jitendra Malik UQCV 75 65 0 26 Sep 2022
A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases James Harrison Luke Metz Jascha Narain Sohl-Dickstein 49 22 0 22 Sep 2022
Improving Meta-Learning Generalization with Activation-Based Early-Stopping Simon Guiroy C. Pal Gonçalo Mordido Sarath Chandar 38 3 0 03 Aug 2022
Gradient-based Bi-level Optimization for Deep Learning: A Survey Can Chen Xiangshan Chen Chen Ma Zixuan Liu Xue Liu 98 35 0 24 Jul 2022
Landscape Learning for Neural Network Inversion Ruoshi Liu Chen-Guang Mao Purva Tendulkar Hongya Wang Carl Vondrick 38 8 0 17 Jun 2022
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities Yisheng Song Ting-Yuan Wang S. Mondal J. P. Sahoo SLR 54 345 0 13 May 2022
Meta-AF: Meta-Learning for Adaptive Filters Jonah Casebeer Nicholas J. Bryan Paris Smaragdis 173 28 0 25 Apr 2022
Expert-Calibrated Learning for Online Optimization with Switching Costs Pengfei Li Jianyi Yang Shaolei Ren 29 11 0 18 Apr 2022
Practical tradeoffs between memory, compute, and performance in learned optimizers Luke Metz C. Freeman James Harrison Niru Maheswaranathan Jascha Narain Sohl-Dickstein 41 32 0 22 Mar 2022
Symbolic Learning to Optimize: Towards Interpretability and Scalability Wenqing Zheng Tianlong Chen Ting-Kuei Hu Zhangyang Wang 45 19 0 13 Mar 2022
Meta Mirror Descent: Optimiser Learning for Fast Convergence Boyan Gao Henry Gouk Haebeom Lee Timothy M. Hospedales 27 6 0 05 Mar 2022
Amortized Proximal Optimization Juhan Bae Paul Vicol Jeff Z. HaoChen Roger C. Grosse ODL 29 14 0 28 Feb 2022
Teaching Networks to Solve Optimization Problems Xinran Liu Yuzhe Lu Ali Abbasi Meiyi Li Javad Mohammadi Soheil Kolouri 44 11 0 08 Feb 2022
Tutorial on amortized optimization Brandon Amos OffRL 78 43 0 01 Feb 2022
Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies Paul Vicol Luke Metz Jascha Narain Sohl-Dickstein 27 68 0 27 Dec 2021
Learning for Robust Combinatorial Optimization: Algorithm and Application Zhihui Shao Jianyi Yang Cong Shen Shaolei Ren 38 6 0 20 Dec 2021
ModelPred: A Framework for Predicting Trained Model from Training Data Yingyan Zeng Jiachen T. Wang Si-An Chen H. Just Ran Jin R. Jia TDI MU 33 2 0 24 Nov 2021
Gradients are Not All You Need Luke Metz C. Freeman S. Schoenholz Tal Kachman 30 93 0 10 Nov 2021
Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection HanQin Cai Jialin Liu W. Yin 38 39 0 11 Oct 2021
Auto-DSP: Learning to Optimize Acoustic Echo Cancellers Jonah Casebeer Nicholas J. Bryan Paris Smaragdis 25 10 0 08 Oct 2021
Benchmarking Safety Monitors for Image Classifiers with Machine Learning Raul Sena Ferreira J. Arlat Jérémie Guiochet H. Waeselynck 43 26 0 04 Oct 2021
Bootstrapped Meta-Learning Sebastian Flennerhag Yannick Schroecker Tom Zahavy Hado van Hasselt David Silver Satinder Singh 38 59 0 09 Sep 2021
Learn2Hop: Learned Optimization on Rough Landscapes Amil Merchant Luke Metz S. Schoenholz E. D. Cubuk 31 16 0 20 Jul 2021
Learning to Optimize: A Primer and A Benchmark Tianlong Chen Xiaohan Chen Wuyang Chen Howard Heaton Jialin Liu Zhangyang Wang W. Yin 54 225 0 23 Mar 2021
How to decay your learning rate Aitor Lewkowycz 41 24 0 23 Mar 2021
Evolving Continuous Optimisers from Scratch M. Lones 27 10 0 22 Mar 2021
Training Learned Optimizers with Randomly Initialized Learned Optimizers Luke Metz C. Freeman Niru Maheswaranathan Jascha Narain Sohl-Dickstein 43 12 0 14 Jan 2021
Reverse engineering learned optimizers reveals known and novel mechanisms Niru Maheswaranathan David Sussillo Luke Metz Ruoxi Sun Jascha Narain Sohl-Dickstein 22 21 0 04 Nov 2020
Training Stronger Baselines for Learning to Optimize Tianlong Chen Weiyi Zhang Jingyang Zhou Shiyu Chang Sijia Liu Lisa Amini Zhangyang Wang OffRL 27 51 0 18 Oct 2020
Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves Luke Metz Niru Maheswaranathan C. Freeman Ben Poole Jascha Narain Sohl-Dickstein 33 62 0 23 Sep 2020
Contextualizing Enhances Gradient Based Meta Learning Evan Vogelbaum Rumen Dangovski L. Jing Marin Soljacic 34 3 0 17 Jul 2020