Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation

20 October 2021

José Miguel Hernández-Lobato

Papers citing "Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation"

39 / 39 papers shown

Title
Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020 Ryan Turner David Eriksson M. McCourt J. Kiili Eero Laaksonen Zhen Xu Isabelle M Guyon BDL 60 300 0 20 Apr 2021
Improved Bilevel Model: Fast and Optimal Algorithm with Theoretical Guarantee Junyi Li Bin Gu Heng-Chiao Huang 66 41 0 01 Sep 2020
Adaptive Hierarchical Hyper-gradient Descent Renlong Jie Junbin Gao A. Vasnev Minh-Ngoc Tran 45 5 0 17 Aug 2020
A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singleton Risheng Liu Pan Mu Xiaoming Yuan Shangzhi Zeng Jin Zhang 58 130 0 07 Jun 2020
Online Hyperparameter Search Interleaved with Proximal Parameter Updates Luis Miguel Lopez Ramos B. Beferull-Lozano 50 3 0 06 Apr 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 499 42,449 0 03 Dec 2019
Penalty Method for Inversion-Free Deep Bilevel Optimization Akshay Mehra Jihun Hamm 90 46 0 08 Nov 2019
Optimizing Millions of Hyperparameters by Implicit Differentiation Jonathan Lorraine Paul Vicol David Duvenaud DD 114 415 0 06 Nov 2019
Generalized Inner Loop Meta-Learning Jaya Kumar Alageshan Brandon Amos A. Verma Phu Mon Htut Artem Molchanov Franziska Meier Douwe Kiela Kyunghyun Cho Soumith Chintala AI4CE 95 159 0 03 Oct 2019
Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning Valerio Perrone Huibin Shen Matthias Seeger Cédric Archambeau Rodolphe Jenatton 60 97 0 27 Sep 2019
Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions M. Mackay Paul Vicol Jonathan Lorraine David Duvenaud Roger C. Grosse 78 164 0 07 Mar 2019
Truncated Back-propagation for Bilevel Optimization Amirreza Shaban Ching-An Cheng Nathan Hatch Byron Boots 101 266 0 25 Oct 2018
A System for Massively Parallel Hyperparameter Tuning Liam Li Kevin Jamieson Afshin Rostamizadeh Ekaterina Gonina Moritz Hardt Benjamin Recht Ameet Talwalkar 66 384 0 13 Oct 2018
Tune: A Research Platform for Distributed Model Selection and Training Richard Liaw Eric Liang Robert Nishihara Philipp Moritz Joseph E. Gonzalez Ion Stoica 209 901 0 13 Jul 2018
BOHB: Robust and Efficient Hyperparameter Optimization at Scale Stefan Falkner Aaron Klein Frank Hutter BDL 203 1,098 0 04 Jul 2018
Bilevel Programming for Hyperparameter Optimization and Meta-Learning Luca Franceschi P. Frasconi Saverio Salzo Riccardo Grazzi Massimiliano Pontil 176 728 0 13 Jun 2018
Reviving and Improving Recurrent Back-Propagation Renjie Liao Yuwen Xiong Ethan Fetaya Lisa Zhang Kijung Yoon Xaq Pitkow R. Urtasun R. Zemel BDL 72 120 0 16 Mar 2018
Understanding Short-Horizon Bias in Stochastic Meta-Optimization Yuhuai Wu Mengye Ren Renjie Liao Roger C. Grosse 99 138 0 06 Mar 2018
Stochastic Hyperparameter Optimization through Hypernetworks Jonathan Lorraine David Duvenaud 74 140 0 26 Feb 2018
Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning Jiazhuo Wang Jason Xu Xuejun Wang 48 73 0 05 Jan 2018
Population Based Training of Neural Networks Max Jaderberg Valentin Dalibard Simon Osindero Wojciech M. Czarnecki Jeff Donahue ... Tim Green Iain Dunning Karen Simonyan Chrisantha Fernando Koray Kavukcuoglu 79 741 0 27 Nov 2017
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms Han Xiao Kashif Rasul Roland Vollgraf 283 8,883 0 25 Aug 2017
Multi-fidelity Bayesian Optimisation with Continuous Approximations Kirthevasan Kandasamy Gautam Dasarathy J. Schneider Barnabás Póczós 60 221 0 18 Mar 2017
Online Learning Rate Adaptation with Hypergradient Descent A. G. Baydin R. Cornish David Martínez-Rubio Mark Schmidt Frank Wood ODL 74 247 0 14 Mar 2017
Forward and Reverse Gradient-Based Hyperparameter Optimization Luca Franceschi Michele Donini P. Frasconi Massimiliano Pontil 213 417 0 06 Mar 2017
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization Lisha Li Kevin Jamieson Giulia DeSalvo Afshin Rostamizadeh Ameet Talwalkar 225 2,329 0 21 Mar 2016
Selecting Near-Optimal Learners via Incremental Data Allocation Ashish Sabharwal Horst Samulowitz Gerald Tesauro 54 64 0 31 Dec 2015
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 194,020 0 10 Dec 2015
Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters Jelena Luketina Mathias Berglund Klaus Greff T. Raiko 55 173 0 20 Nov 2015
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning Y. Gal Zoubin Ghahramani UQCV BDL 821 9,318 0 06 Jun 2015
Non-stochastic Best Arm Identification and Hyperparameter Optimization Kevin Jamieson Ameet Talwalkar 206 580 0 27 Feb 2015
Scalable Bayesian Optimization Using Deep Neural Networks Jasper Snoek Oren Rippel Kevin Swersky Ryan Kiros N. Satish N. Sundaram Md. Mostofa Ali Patwary P. Prabhat Ryan P. Adams BDL UQCV 95 1,043 0 19 Feb 2015
Gradient-based Hyperparameter Optimization through Reversible Learning D. Maclaurin David Duvenaud Ryan P. Adams DD 227 945 0 11 Feb 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 1.8K 150,115 0 22 Dec 2014
Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces Kevin Swersky David Duvenaud Jasper Snoek Frank Hutter Michael A. Osborne BDL 145 79 0 14 Sep 2014
Freeze-Thaw Bayesian Optimization Kevin Swersky Jasper Snoek Ryan P. Adams 84 269 0 16 Jun 2014
Bayesian Optimization in a Billion Dimensions via Random Embeddings Ziyun Wang Frank Hutter M. Zoghi David Matheson Nando de Freitas 165 443 0 09 Jan 2013
Practical Bayesian Optimization of Machine Learning Algorithms Jasper Snoek Hugo Larochelle Ryan P. Adams 356 7,942 0 13 Jun 2012
Fast Cross-Validation via Sequential Testing Tammo Krueger Danny Panknin M. Braun 80 51 0 11 Jun 2012