Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design

22 May 2023

Ibrahim Alabdulmohsin

Papers citing "Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design"

50 / 57 papers shown

Title
Inference Compute-Optimal Video Vision Language Models Peiqi Wang ShengYun Peng Xuewen Zhang Hanchao Yu Yibo Yang Lifu Huang Fujun Liu Qifan Wang VLM 90 0 0 24 May 2025
Stronger ViTs With Octic Equivariance David Nordström Johan Edstedt Fredrik Kahl Georg Bökman ViT 211 0 0 21 May 2025
SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning Yang Liu Ming Ma Xiaomin Yu Pengxiang Ding Han Zhao Mingyang Sun Siteng Huang Donglin Wang LRM 158 0 0 18 May 2025
DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs Zehao Wang Senthil Purushwalkam Caiming Xiong Siyang Song Chenhui Xu Ran Xu 134 2 0 23 Apr 2025
Mixtera: A Data Plane for Foundation Model Training Maximilian Böther Xiaozhe Yao Tolga Kerimoglu Ana Klimovic Viktor Gsteiger Ana Klimovic MoE 155 0 0 27 Feb 2025
Simpler Fast Vision Transformers with a Jumbo CLS Token A. Fuller Yousef Yassin Daniel G. Kyrollos Evan Shelhamer James R. Green 147 0 0 20 Feb 2025
jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images Andreas Koukounas Georgios Mastrapas Bo Wang Mohammad Kalim Akram Sedigheh Eslami Michael Gunther Isabelle Mohr Saba Sturua Scott Martens Nan Wang VLM 296 9 0 11 Dec 2024
TIPS: Text-Image Pretraining with Spatial awareness Kevis-Kokitsi Maninis Kaifeng Chen Soham Ghosh Arjun Karpur Koert Chen ... Jan Dlabal Dan Gnanapragasam Mojtaba Seyedhosseini Howard Zhou Andre Araujo VLM 80 3 0 21 Oct 2024
Locality Alignment Improves Vision-Language Models Ian Covert Tony Sun James Zou Tatsunori Hashimoto VLM 231 6 0 14 Oct 2024
Scaling Optimal LR Across Token Horizons Johan Bjorck Alon Benhaim Vishrav Chaudhary Furu Wei Xia Song 97 6 0 30 Sep 2024
Resolving Discrepancies in Compute-Optimal Scaling of Language Models Tomer Porian Mitchell Wortsman J. Jitsev Ludwig Schmidt Y. Carmon 110 24 0 27 Jun 2024
ColPali: Efficient Document Retrieval with Vision Language Models Manuel Faysse Hugues Sibille Tony Wu Bilel Omrani Gautier Viaud C´eline Hudelot Pierre Colombo VLM 233 26 0 27 Jun 2024
Multi-objective Differentiable Neural Architecture Search R. Sukthanker Arber Zela B. Staffler Samuel Dooley Josif Grabocka Frank Hutter 92 1 0 28 Feb 2024
A Study of Autoregressive Decoders for Multi-Tasking in Computer Vision Lucas Beyer Bo Wan Gagan Madan Filip Pavetić Andreas Steiner ... Emanuele Bugliarello Tianlin Li Qihang Yu Liang-Chieh Chen Xiaohua Zhai 100 8 0 30 Mar 2023
FlexiViT: One Model for All Patch Sizes Lucas Beyer Pavel Izmailov Alexander Kolesnikov Mathilde Caron Simon Kornblith Xiaohua Zhai Matthias Minderer Michael Tschannen Ibrahim Alabdulmohsin Filip Pavetić VLM 100 94 0 15 Dec 2022
PaLI: A Jointly-Scaled Multilingual Language-Image Model Xi Chen Tianlin Li Soravit Changpinyo A. Piergiovanni Piotr Padlewski ... Andreas Steiner A. Angelova Xiaohua Zhai N. Houlsby Radu Soricut MLLM VLM 92 720 0 14 Sep 2022
Revisiting Neural Scaling Laws in Language and Vision Ibrahim Alabdulmohsin Behnam Neyshabur Xiaohua Zhai 204 109 0 13 Sep 2022
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? Yi Tay Mostafa Dehghani Samira Abnar Hyung Won Chung W. Fedus J. Rao Sharan Narang Vinh Q. Tran Dani Yogatama Donald Metzler AI4CE 95 104 0 21 Jul 2022
UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes Alexander Kolesnikov André Susano Pinto Lucas Beyer Xiaohua Zhai Jeremiah Harmsen N. Houlsby 153 70 0 20 May 2022
DeiT III: Revenge of the ViT Hugo Touvron Matthieu Cord Hervé Jégou ViT 120 413 0 14 Apr 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 480 6,240 0 05 Apr 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 203 1,946 0 29 Mar 2022
Florence: A New Foundation Model for Computer Vision Lu Yuan Dongdong Chen Yi-Ling Chen Noel Codella Xiyang Dai ... Zhen Xiao Jianwei Yang Michael Zeng Luowei Zhou Pengchuan Zhang VLM 130 904 0 22 Nov 2021
Combined Scaling for Zero-shot Transfer Learning Hieu H. Pham Zihang Dai Golnaz Ghiasi Kenji Kawaguchi Hanxiao Liu ... Yi-Ting Chen Minh-Thang Luong Yonghui Wu Mingxing Tan Quoc V. Le VLM 63 199 0 19 Nov 2021
Swin Transformer V2: Scaling Up Capacity and Resolution Ze Liu Han Hu Yutong Lin Zhuliang Yao Zhenda Xie ... Yue Cao Zheng Zhang Li Dong Furu Wei B. Guo ViT 209 1,813 0 18 Nov 2021
LiT: Zero-Shot Transfer with Locked-image text Tuning Xiaohua Zhai Tianlin Li Basil Mustafa Andreas Steiner Daniel Keysers Alexander Kolesnikov Lucas Beyer VLM 95 556 0 15 Nov 2021
The Efficiency Misnomer Daoyuan Chen Liuyi Yao Dawei Gao Ashish Vaswani Yaliang Li 93 103 0 25 Oct 2021
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers Yi Tay Mostafa Dehghani J. Rao W. Fedus Samira Abnar Hyung Won Chung Sharan Narang Dani Yogatama Ashish Vaswani Donald Metzler 242 114 0 22 Sep 2021
Scaling Laws for Neural Machine Translation Behrooz Ghorbani Orhan Firat Markus Freitag Ankur Bapna M. Krikun Xavier Garcia Ciprian Chelba Colin Cherry 69 102 0 16 Sep 2021
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers Andreas Steiner Alexander Kolesnikov Xiaohua Zhai Ross Wightman Jakob Uszkoreit Lucas Beyer ViT 107 633 0 18 Jun 2021
CoAtNet: Marrying Convolution and Attention for All Data Sizes Zihang Dai Hanxiao Liu Quoc V. Le Mingxing Tan ViT 114 1,204 0 09 Jun 2021
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 333 671 0 21 Apr 2021
Revisiting ResNets: Improved Training and Scaling Strategies Irwan Bello W. Fedus Xianzhi Du E. D. Cubuk A. Srinivas Nayeon Lee Jonathon Shlens Barret Zoph 84 300 0 13 Mar 2021
Explaining Neural Scaling Laws Yasaman Bahri Ethan Dyer Jared Kaplan Jaehoon Lee Utkarsh Sharma 62 263 0 12 Feb 2021
Learning Curve Theory Marcus Hutter 202 63 0 08 Feb 2021
Scaling Laws for Transfer Danny Hernandez Jared Kaplan T. Henighan Sam McCandlish 80 245 0 02 Feb 2021
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization Dan Hendrycks Steven Basart Norman Mu Saurav Kadavath Frank Wang ... Samyak Parajuli Mike Guo D. Song Jacob Steinhardt Justin Gilmer OOD 328 1,740 0 29 Jun 2020
Are we done with ImageNet? Lucas Beyer Olivier J. Hénaff Alexander Kolesnikov Xiaohua Zhai Aaron van den Oord VLM 119 401 0 12 Jun 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 602 4,822 0 23 Jan 2020
Big Transfer (BiT): General Visual Representation Learning Alexander Kolesnikov Lucas Beyer Xiaohua Zhai J. Puigcerver Jessica Yung Sylvain Gelly N. Houlsby MQ 286 1,205 0 24 Dec 2019
Natural Adversarial Examples Dan Hendrycks Kevin Zhao Steven Basart Jacob Steinhardt D. Song OODD 203 1,472 0 16 Jul 2019
Fixing the train-test resolution discrepancy Hugo Touvron Andrea Vedaldi Matthijs Douze Hervé Jégou 116 423 0 14 Jun 2019
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks Mingxing Tan Quoc V. Le 3DV MedIm 139 18,134 0 28 May 2019
Do ImageNet Classifiers Generalize to ImageNet? Benjamin Recht Rebecca Roelofs Ludwig Schmidt Vaishaal Shankar OOD SSeg VLM 113 1,715 0 13 Feb 2019
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost Noam M. Shazeer Mitchell Stern ODL 76 1,048 0 11 Apr 2018
MobileNetV2: Inverted Residuals and Linear Bottlenecks Mark Sandler Andrew G. Howard Menglong Zhu A. Zhmoginov Liang-Chieh Chen 181 19,284 0 13 Jan 2018
Deep Learning Scaling is Predictable, Empirically Joel Hestness Sharan Narang Newsha Ardalani G. Diamos Heewoo Jun Hassan Kianinejad Md. Mostofa Ali Patwary Yang Yang Yanqi Zhou 92 741 0 01 Dec 2017
Neural Discrete Representation Learning Aaron van den Oord Oriol Vinyals Koray Kavukcuoglu BDL SSL OCL 226 5,019 0 02 Nov 2017
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era Chen Sun Abhinav Shrivastava Saurabh Singh Abhinav Gupta VLM 185 2,398 0 10 Jul 2017
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand M. Andreetto Hartwig Adam 3DH 1.1K 20,837 0 17 Apr 2017