Using a thousand optimization tasks to learn hyperparameter search strategies

27 February 2020

Jascha Narain Sohl-Dickstein

ArXiv PDF HTML

Papers citing "Using a thousand optimization tasks to learn hyperparameter search strategies"

50 / 79 papers shown

Title
Decision-Making with Auto-Encoding Variational Bayes Romain Lopez Pierre Boyeau Nir Yosef Michael I. Jordan Jeffrey Regier BDL 142 10,591 0 17 Feb 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 160 42,038 0 03 Dec 2019
Leveraging Procedural Generation to Benchmark Reinforcement Learning K. Cobbe Christopher Hesse Jacob Hilton John Schulman 52 549 0 03 Dec 2019
Optimizer Benchmarking Needs to Account for Hyperparameter Tuning Prabhu Teja Sivaprasad Florian Mai Thijs Vogels Martin Jaggi François Fleuret 34 12 0 25 Oct 2019
Solving Rubik's Cube with a Robot Hand OpenAI Ilge Akkaya Marcin Andrychowicz Maciek Chociej Ma-teusz Litwin ... Peter Welinder Lilian Weng Qiming Yuan Wojciech Zaremba Lei Zhang ODL 53 1,215 0 16 Oct 2019
On Empirical Comparisons of Optimizers for Deep Learning Dami Choi Christopher J. Shallue Zachary Nado Jaehoon Lee Chris J. Maddison George E. Dahl 53 259 0 11 Oct 2019
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark Xiaohua Zhai J. Puigcerver Alexander Kolesnikov P. Ruyssen C. Riquelme ... Michael Tschannen Marcin Michalski Olivier Bousquet Sylvain Gelly N. Houlsby SSL 49 432 0 01 Oct 2019
Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning Valerio Perrone Huibin Shen Matthias Seeger Cédric Archambeau Rodolphe Jenatton 44 96 0 27 Sep 2019
On the Variance of the Adaptive Learning Rate and Beyond Liyuan Liu Haoming Jiang Pengcheng He Weizhu Chen Xiaodong Liu Jianfeng Gao Jiawei Han ODL 115 1,894 0 08 Aug 2019
Using learned optimizers to make models robust to input noise Luke Metz Niru Maheswaranathan Jonathon Shlens Jascha Narain Sohl-Dickstein E. D. Cubuk VLM OOD 29 26 0 08 Jun 2019
Energy and Policy Considerations for Deep Learning in NLP Emma Strubell Ananya Ganesh Andrew McCallum 38 2,633 0 05 Jun 2019
A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off Yaniv Blumenfeld D. Gilboa Daniel Soudry MQ 32 14 0 03 Jun 2019
Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization Aaron Klein Frank Hutter 21 92 0 13 May 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems Alex Jinpeng Wang Yada Pruksachatkun Nikita Nangia Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 157 2,287 0 02 May 2019
DeepOBS: A Deep Learning Optimizer Benchmark Suite Frank Schneider Lukas Balles Philipp Hennig ODL 92 71 0 13 Mar 2019
Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples Eleni Triantafillou Tyler Lixuan Zhu Vincent Dumoulin Pascal Lamblin Utku Evci ... Ross Goroshin Carles Gelada Kevin Swersky Pierre-Antoine Manzagol Hugo Larochelle 109 614 0 07 Mar 2019
NAS-Bench-101: Towards Reproducible Neural Architecture Search Chris Ying Aaron Klein Esteban Real Eric Christiansen Kevin Patrick Murphy Frank Hutter 39 678 0 25 Feb 2019
Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning Arthur Juliani Ahmed Khalifa Vincent-Pierre Berges Jonathan Harper Ervin Teng Hunter Henry A. Crespi Julian Togelius Danny Lange 29 143 0 04 Feb 2019
Open-ended Learning in Symmetric Zero-sum Games David Balduzzi M. Garnelo Yoram Bachrach Wojciech M. Czarnecki Julien Perolat Max Jaderberg T. Graepel 35 169 0 23 Jan 2019
Bayesian Optimization in AlphaGo Yutian Chen Aja Huang Ziyun Wang Ioannis Antonoglou Julian Schrittwieser David Silver Nando de Freitas BDL 39 113 0 17 Dec 2018
An Empirical Model of Large-Batch Training Sam McCandlish Jared Kaplan Dario Amodei OpenAI Dota Team 54 275 0 14 Dec 2018
Quantifying Generalization in Reinforcement Learning K. Cobbe Oleg Klimov Christopher Hesse Taehoon Kim John Schulman OffRL 68 662 0 06 Dec 2018
Learning Multiple Defaults for Machine Learning Algorithms Florian Pfisterer Jan N. van Rijn Philipp Probst Andreas Müller B. Bischl 27 26 0 23 Nov 2018
Measuring the Effects of Data Parallelism on Neural Network Training Christopher J. Shallue Jaehoon Lee J. Antognini J. Mamou J. Ketterling Yao Wang 63 408 0 08 Nov 2018
Critical initialisation for deep signal propagation in noisy rectifier neural networks Arnu Pretorius Elan Van Biljon Steve Kroon Herman Kamper 26 16 0 01 Nov 2018
Understanding and correcting pathologies in the training of learned optimizers Luke Metz Niru Maheswaranathan Jeremy Nixon C. Freeman Jascha Narain Sohl-Dickstein ODL 48 148 0 24 Oct 2018
DeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learning A. Olsen D. Konovalov B. Philippa P. Ridd Jake Wood ... Owen Kenny J. Whinney Brendan Calvert M. R. Azghadi Ronald D. White 28 378 0 09 Oct 2018
BOHB: Robust and Efficient Hyperparameter Optimization at Scale Stefan Falkner Aaron Klein Frank Hutter BDL 134 1,077 0 04 Jul 2018
The Natural Language Decathlon: Multitask Learning as Question Answering Bryan McCann N. Keskar Caiming Xiong R. Socher AIMat MLLM BDL 56 641 0 20 Jun 2018
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks Lechao Xiao Yasaman Bahri Jascha Narain Sohl-Dickstein S. Schoenholz Jeffrey Pennington 284 353 0 14 Jun 2018
Re-evaluating Evaluation David Balduzzi K. Tuyls Julien Perolat T. Graepel MoMe 44 97 0 07 Jun 2018
Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach Ryo Karakida S. Akaho S. Amari FedML 104 142 0 04 Jun 2018
Parallel Architecture and Hyperparameter Search via Successive Halving and Classification Manoj Kumar George E. Dahl Vijay Vasudevan Mohammad Norouzi 46 25 0 25 May 2018
On the Selection of Initialization and Activation Function for Deep Neural Networks Soufiane Hayou Arnaud Doucet Judith Rousseau ODL 33 75 0 21 May 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 570 7,080 0 20 Apr 2018
Gotta Learn Fast: A New Benchmark for Generalization in RL Alex Nichol Vicki Pfau Christopher Hesse Oleg Klimov John Schulman VLM OffRL 29 177 0 10 Apr 2018
Understanding Short-Horizon Bias in Stochastic Meta-Optimization Yuhuai Wu Mengye Ren Renjie Liao Roger C. Grosse 61 137 0 06 Mar 2018
DeepMind Control Suite Yuval Tassa Yotam Doron Alistair Muldal Tom Erez Yazhe Li ... A. Abdolmaleki J. Merel Andrew Lefrancq Timothy Lillicrap Martin Riedmiller ELM LM&Ro BDL 97 1,116 0 02 Jan 2018
Mean Field Residual Networks: On the Edge of Chaos Greg Yang S. Schoenholz 26 189 0 24 Dec 2017
Decoupled Weight Decay Regularization I. Loshchilov Frank Hutter OffRL 84 2,112 0 14 Nov 2017
Neural Optimizer Search with Reinforcement Learning Irwan Bello Barret Zoph Vijay Vasudevan Quoc V. Le ODL 42 385 0 21 Sep 2017
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms Han Xiao Kashif Rasul Roland Vollgraf 110 8,807 0 25 Aug 2017
SMASH: One-Shot Model Architecture Search through HyperNetworks Andrew Brock Theodore Lim J. Ritchie Nick Weston 79 762 0 17 Aug 2017
A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets P. Chrabaszcz I. Loshchilov Frank Hutter SSeg OOD 86 640 0 27 Jul 2017
On the State of the Art of Evaluation in Neural Language Models Gábor Melis Chris Dyer Phil Blunsom 42 532 0 18 Jul 2017
Emergence of Locomotion Behaviours in Rich Environments N. Heess TB Dhruva S. Sriram Jay Lemmon J. Merel ... Tom Erez Ziyun Wang S. M. Ali Eslami Martin Riedmiller David Silver 183 931 0 07 Jul 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 333 129,831 0 12 Jun 2017
Critical Hyper-Parameters: No Random, No Cry Olivier Bousquet Sylvain Gelly Karol Kurach O. Teytaud Damien Vincent 21 40 0 10 Jun 2017
The Marginal Value of Adaptive Gradient Methods in Machine Learning Ashia Wilson Rebecca Roelofs Mitchell Stern Nathan Srebro Benjamin Recht ODL 44 1,023 0 23 May 2017
Masked Autoregressive Flow for Density Estimation George Papamakarios Theo Pavlakou Iain Murray 111 1,340 0 19 May 2017