VeLO: Training Versatile Learned Optimizers by Scaling Up

17 November 2022

Jascha Narain Sohl-Dickstein

ArXiv PDF HTML

Papers citing "VeLO: Training Versatile Learned Optimizers by Scaling Up"

50 / 51 papers shown

Title
Recursive Self-Similarity in Deep Weight Spaces of Neural Architectures: A Fractal and Coarse Geometry Perspective A. Moharil I. Kumara Damian Tamburri Majid Mohammadi Willem-jan Van Den Heuvel 60 0 0 18 Mar 2025
Make Optimization Once and for All with Fine-grained Guidance Mingjia Shi Ruihan Lin Xuxi Chen Yuhao Zhou Zezhen Ding ... Tong Wang Kai Wang Zhangyang Wang J. Zhang Tianlong Chen 53 1 0 14 Mar 2025
Learning Versatile Optimizers on a Compute Diet A. Moudgil Boris Knyazev Guillaume Lajoie Eugene Belilovsky 138 0 0 22 Jan 2025
Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning Richa Upadhyay Ronald Phlypo Rajkumar Saini Marcus Liwicki 35 0 0 21 Jan 2025
$Applications of fractional calculus in learned optimization$ Applications of fractional calculus in learned optimization Teodor Alexandru Szente James Harrison M. Zanfir C. Sminchisescu 66 0 0 22 Nov 2024
Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models Theo Putterman Derek Lim Yoav Gelberg Stefanie Jegelka Haggai Maron AI4CE 43 5 0 05 Oct 2024
Narrowing the Focus: Learned Optimizers for Pretrained Models Gus Kristiansen Mark Sandler A. Zhmoginov Nolan Miller Anirudh Goyal Jihwan Lee Max Vladymyrov 34 1 0 17 Aug 2024
Learning to Explore for Stochastic Gradient MCMC Seunghyun Kim Seohyeon Jung Seonghyeon Kim Juho Lee BDL 48 1 0 17 Aug 2024
Learning to Learn without Forgetting using Attention Anna Vettoruzzo Joaquin Vanschoren Mohamed-Rafik Bouguelia Thorsteinn Rögnvaldsson CLL 39 2 0 06 Aug 2024
Can Learned Optimization Make Reinforcement Learning Less Difficult? Alexander David Goldie Chris Xiaoxuan Lu Matthew Jackson Shimon Whiteson Jakob N. Foerster 40 3 0 09 Jul 2024
Discovering Minimal Reinforcement Learning Environments Jarek Liesen Chris Xiaoxuan Lu Andrei Lupu Jakob N. Foerster Henning Sprekeler R. T. Lange OffRL 46 3 0 18 Jun 2024
Discovering Preference Optimization Algorithms with and for Large Language Models Chris Xiaoxuan Lu Samuel Holt Claudio Fanconi Alex J. Chan Jakob Foerster M. Schaar R. T. Lange OffRL 32 15 0 12 Jun 2024
$μ$ LO: Compute-Efficient Meta-Generalization of Learned Optimizers Benjamin Thérien Charles-Étienne Joseph Boris Knyazev Edouard Oyallon Irina Rish Eugene Belilovsky AI4CE 38 1 0 31 May 2024
Implicit Neural Image Field for Biological Microscopy Image Compression Gaole Dai Cheng-Ching Tseng Qingpo Wuwu Rongyu Zhang Shaokang Wang ... Yu Zhou A. A. Tuz Matthias Gunzer Jianxu Chen Shanghang Zhang 25 1 0 29 May 2024
Text-to-Model: Text-Conditioned Neural Network Diffusion for Train-Once-for-All Personalization Zexi Li Lingzhi Gao Chao Wu AI4CE DiffM 55 3 0 23 May 2024
Graph Neural Networks for Learning Equivariant Representations of Neural Networks Miltiadis Kofinas Boris Knyazev Yan Zhang Yunlu Chen Gertjan J. Burghouts E. Gavves Cees G. M. Snoek David W. Zhang 44 29 0 18 Mar 2024
Dynamic Memory Based Adaptive Optimization Balázs Szegedy Domonkos Czifra Péter Korösi-Szabó ODL 27 0 0 23 Feb 2024
Discovering Temporally-Aware Reinforcement Learning Algorithms Matthew Jackson Chris Xiaoxuan Lu Louis Kirsch R. T. Lange Shimon Whiteson Jakob N. Foerster 19 18 0 08 Feb 2024
Universal Neural Functionals Allan Zhou Chelsea Finn James Harrison 27 12 0 07 Feb 2024
AutoFT: Learning an Objective for Robust Fine-Tuning Caroline Choi Yoonho Lee Annie S. Chen Allan Zhou Aditi Raghunathan Chelsea Finn OOD 44 0 0 18 Jan 2024
MADA: Meta-Adaptive Optimizers through hyper-gradient Descent Kaan Ozkara Can Karakus Parameswaran Raman Mingyi Hong Shoham Sabach B. Kveton V. Cevher 21 2 0 17 Jan 2024
Graph Metanetworks for Processing Diverse Neural Architectures Derek Lim Haggai Maron Marc T. Law Jonathan Lorraine James Lucas AI4CE 31 30 0 07 Dec 2023
Can We Learn Communication-Efficient Optimizers? Charles-Étienne Joseph Benjamin Thérien A. Moudgil Boris Knyazev Eugene Belilovsky 29 1 0 02 Dec 2023
Generalisable Agents for Neural Network Optimisation Kale-ab Tessera C. Tilbury Sasha Abramowitz Ruan de Kock Omayma Mahjoub Benjamin Rosman Sara Hooker Arnu Pretorius AI4CE 20 0 0 30 Nov 2023
A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors Olivier Laurent Emanuel Aldea Gianni Franchi BDL UQCV 20 5 0 12 Oct 2023
Making Scalable Meta Learning Practical Sang Keun Choe Sanket Vaibhav Mehta Hwijeen Ahn W. Neiswanger Pengtao Xie Emma Strubell Eric P. Xing 47 14 0 09 Oct 2023
A Machine Learning-oriented Survey on Tiny Machine Learning Luigi Capogrosso Federico Cunico D. Cheng Franco Fummi Marco Cristani SyDa MU 29 33 0 21 Sep 2023
Learning to Warm-Start Fixed-Point Optimization Algorithms Rajiv Sambharya Georgina Hall Brandon Amos Bartolomeo Stellato 30 12 0 14 Sep 2023
Hyperparameters in Reinforcement Learning and How To Tune Them Theresa Eimer Marius Lindauer Roberta Raileanu OffRL 27 34 0 02 Jun 2023
HUB: Guiding Learned Optimizers with Continuous Prompt Tuning Gaole Dai Wei Yu Wu Ziyu Wang Jie Fu Shanghang Zhang Tiejun Huang AIFin 14 0 0 26 May 2023
Neural Functional Transformers Allan Zhou Kaien Yang Yiding Jiang Kaylee Burns Winnie Xu Samuel Sokota J. Zico Kolter Chelsea Finn 16 31 0 22 May 2023
Graph Reinforcement Learning for Network Control via Bi-Level Optimization Daniele Gammelli James Harrison Kaidi Yang Marco Pavone Filipe Rodrigues Francisco Câmara Pereira AI4CE 28 6 0 16 May 2023
Discovering Attention-Based Genetic Algorithms via Meta-Black-Box Optimization R. T. Lange Tom Schaul Yutian Chen Chris Xiaoxuan Lu Tom Zahavy Valentin Dalibard Sebastian Flennerhag 24 34 0 08 Apr 2023
Mathematical Challenges in Deep Learning V. Nia Guojun Zhang I. Kobyzev Michael R. Metel Xinlin Li ... S. Hemati M. Asgharian Linglong Kong Wulong Liu Boxing Chen AI4CE VLM 37 1 0 24 Mar 2023
Permutation Equivariant Neural Functionals Allan Zhou Kaien Yang Kaylee Burns Adriano Cardace Yiding Jiang Samuel Sokota J. Zico Kolter Chelsea Finn 25 47 0 27 Feb 2023
MONGOOSE: Path-wise Smooth Bayesian Optimisation via Meta-learning Adam X. Yang Laurence Aitchison Henry B. Moss 29 4 0 22 Feb 2023
Symbolic Discovery of Optimization Algorithms Xiangning Chen Chen Liang Da Huang Esteban Real Kaiyuan Wang ... Xuanyi Dong Thang Luong Cho-Jui Hsieh Yifeng Lu Quoc V. Le 55 350 0 13 Feb 2023
Learning to Optimize for Reinforcement Learning Qingfeng Lan Rupam Mahmood Shuicheng Yan Zhongwen Xu OffRL 24 6 0 03 Feb 2023
Mnemosyne: Learning to Train Transformers with Transformers Deepali Jain K. Choromanski Kumar Avinava Dubey Sumeet Singh Vikas Sindhwani Tingnan Zhang Jie Tan OffRL 33 9 0 02 Feb 2023
A Nonstochastic Control Approach to Optimization Xinyi Chen Elad Hazan 47 5 0 19 Jan 2023
Federated Automatic Differentiation Keith Rush Zachary B. Charles Zachary Garrett FedML 34 1 0 18 Jan 2023
A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases James Harrison Luke Metz Jascha Narain Sohl-Dickstein 44 22 0 22 Sep 2022
Meta Optimal Transport Brandon Amos Samuel N. Cohen Giulia Luise I. Redko OT 22 22 0 10 Jun 2022
Tutorial on amortized optimization Brandon Amos OffRL 75 43 0 01 Feb 2022
A Simple Guard for Learned Optimizers Isabeau Prémont-Schwarz Jaroslav Vítkru Jan Feyereisl 49 7 0 28 Jan 2022
Primer: Searching for Efficient Transformers for Language Modeling David R. So Wojciech Mañke Hanxiao Liu Zihang Dai Noam M. Shazeer Quoc V. Le VLM 85 152 0 17 Sep 2021
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 271 2,603 0 04 May 2021
Training Learned Optimizers with Randomly Initialized Learned Optimizers Luke Metz C. Freeman Niru Maheswaranathan Jascha Narain Sohl-Dickstein 41 12 0 14 Jan 2021
On the distance between two neural networks and the stability of learning Jeremy Bernstein Arash Vahdat Yisong Yue Ming-Yu Liu ODL 195 57 0 09 Feb 2020
CAD2RL: Real Single-Image Flight without a Single Real Image Fereshteh Sadeghi Sergey Levine SSL 226 809 0 13 Nov 2016