Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization

Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization

14 April 2025

Timur Carstensen

Neeratyoy Mallik

Papers citing "Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization"

19 / 19 papers shown

Title
Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization Mohammad Samragh Iman Mirzadeh Keivan Alizadeh Vahid Fartash Faghri Minsik Cho Moin Nabi Devang Naik Mehrdad Farajtabar LRM AI4CE 38 7 0 19 Sep 2024
Resolving Discrepancies in Compute-Optimal Scaling of Language Models Tomer Porian Mitchell Wortsman J. Jitsev Ludwig Schmidt Y. Carmon 95 23 0 27 Jun 2024
Automated Dynamic Algorithm Configuration Steven Adriaensen André Biedenkapp Gresa Shala Noor H. Awad Theresa Eimer Marius Lindauer Frank Hutter 77 37 0 27 May 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 123 1,915 0 29 Mar 2022
Staged Training for Transformer Language Models Sheng Shen Pete Walsh Kurt Keutzer Jesse Dodge Matthew E. Peters Iz Beltagy 32 35 0 11 Mar 2022
ResNet strikes back: An improved training procedure in timm Ross Wightman Hugo Touvron Hervé Jégou AI4TS 230 489 0 01 Oct 2021
Revisiting ResNets: Improved Training and Scaling Strategies Irwan Bello W. Fedus Xianzhi Du E. D. Cubuk A. Srinivas Nayeon Lee Jonathon Shlens Barret Zoph 60 299 0 13 Mar 2021
HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation Alexander I. Cowen-Rivers Wenlong Lyu Rasul Tutunov Zhi Wang Antoine Grosnit ... A. Maraval Hao Jianye Jun Wang Jan Peters H. Ammar 52 74 0 07 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 451 4,662 0 23 Jan 2020
Optimizing Millions of Hyperparameters by Implicit Differentiation Jonathan Lorraine Paul Vicol David Duvenaud DD 104 409 0 06 Nov 2019
Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly Kirthevasan Kandasamy Karun Raju Vysyaraju Willie Neiswanger Biswajit Paria Christopher R. Collins J. Schneider Barnabás Póczós Eric Xing 50 174 0 15 Mar 2019
Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions M. Mackay Paul Vicol Jonathan Lorraine David Duvenaud Roger C. Grosse 72 164 0 07 Mar 2019
A Tutorial on Bayesian Optimization P. Frazier GP 83 1,770 0 08 Jul 2018
BOHB: Robust and Efficient Hyperparameter Optimization at Scale Stefan Falkner Aaron Klein Frank Hutter BDL 160 1,077 0 04 Jul 2018
DARTS: Differentiable Architecture Search Hanxiao Liu Karen Simonyan Yiming Yang 167 4,326 0 24 Jun 2018
FreezeOut: Accelerate Training by Progressively Freezing Layers Andrew Brock Theodore Lim J. Ritchie Nick Weston 37 123 0 15 Jun 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 453 129,831 0 12 Jun 2017
Non-stochastic Best Arm Identification and Hyperparameter Optimization Kevin Jamieson Ameet Talwalkar 146 570 0 27 Feb 2015
Practical Bayesian Optimization of Machine Learning Algorithms Jasper Snoek Hugo Larochelle Ryan P. Adams 292 7,883 0 13 Jun 2012