Automated Progressive Learning for Efficient Training of Vision Transformers

28 March 2022

Changlin Li

Bohan Zhuang

Guangrun Wang

Xiaodan Liang

Xiaojun Chang

Yi Yang

ArXiv PDF HTML

Papers citing "Automated Progressive Learning for Efficient Training of Vision Transformers"

36 / 36 papers shown

Title
STEP: Staged Parameter-Efficient Pre-training for Large Language Models Kazuki Yano Takumi Ito Jun Suzuki LRM 47 1 0 05 Apr 2025
Accelerating Augmentation Invariance Pretraining Jinhong Lin Cheng-En Wu Yibing Wei Pedro Morgado ViT 30 1 0 27 Oct 2024
Efficient Training of Large Vision Models via Advanced Automated Progressive Learning Changlin Li Jiawei Zhang Sihao Lin Zongxin Yang Junwei Liang Xiaodan Liang Xiaojun Chang VLM 23 0 0 06 Sep 2024
ScreenMark: Watermarking Arbitrary Visual Content on Screen Xiujian Liang Gaozhi Liu Yichao Si Xiaoxiao Hu Zhenxing Qian Xinpeng Zhang WIGM 26 0 0 05 Sep 2024
Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Training Sunwoo Lee Tuo Zhang Saurav Prakash Yue Niu Salman Avestimehr FedML 35 4 0 21 Jun 2024
EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Training Yulin Wang Yang Yue Rui Lu Yizeng Han Shiji Song Gao Huang VLM 56 12 0 14 May 2024
Data-independent Module-aware Pruning for Hierarchical Vision Transformers Yang He Joey Tianyi Zhou ViT 47 3 0 21 Apr 2024
FedTrans: Efficient Federated Learning via Multi-Model Transformation Yuxuan Zhu Jiachen Liu Mosharaf Chowdhury Fan Lai 38 0 0 21 Apr 2024
A General and Efficient Training for Transformer via Token Expansion Wenxuan Huang Yunhang Shen Jiao Xie Baochang Zhang Gaoqi He Ke Li Xing Sun Shaohui Lin 42 3 0 31 Mar 2024
Preparing Lessons for Progressive Training on Language Models Yu Pan Ye Yuan Yichun Yin Jiaxin Shi Zenglin Xu Ming Zhang Lifeng Shang Xin Jiang Qun Liu 18 9 0 17 Jan 2024
When To Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks Haihang Wu Wei Wang T. Malepathirana Damith A. Senanayake D. Oetomo Saman K. Halgamuge 18 2 0 06 Jan 2024
Navigating Scaling Laws: Compute Optimality in Adaptive Model Training Sotiris Anagnostidis Gregor Bachmann Imanol Schlag Thomas Hofmann 33 2 0 06 Nov 2023
Reusing Pretrained Models by Multi-linear Operators for Efficient Training Yu Pan Ye Yuan Yichun Yin Zenglin Xu Lifeng Shang Xin Jiang Qun Liu 40 16 0 16 Oct 2023
Computation-efficient Deep Learning for Computer Vision: A Survey Yulin Wang Yizeng Han Chaofei Wang Shiji Song Qi Tian Gao Huang VLM 31 20 0 27 Aug 2023
GrowCLIP: Data-aware Automatic Model Growing for Large-scale Contrastive Language-Image Pre-training Xi Deng Han Shi Runhu Huang Changlin Li Hang Xu Jianhua Han James T. Kwok Shen Zhao Wei Zhang Xiaodan Liang CLIP VLM 29 3 0 22 Aug 2023
E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning Cheng Han Qifan Wang Yiming Cui Zhiwen Cao Wenguan Wang Siyuan Qi Dongfang Liu VPVLM VLM 17 46 0 25 Jul 2023
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models Jean Kaddour Oscar Key Piotr Nawrot Pasquale Minervini Matt J. Kusner 20 41 0 12 Jul 2023
Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity Zijiao Chen Jiaxin Qing J. Zhou DiffM VGen 26 54 0 19 May 2023
Masked Structural Growth for 2x Faster Language Model Pre-training Yiqun Yao Zheng-Wei Zhang Jing Li Yequan Wang OffRL AI4CE LRM 40 15 0 04 May 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 30 40 0 07 Apr 2023
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis Guangcong Wang Zhaoxi Chen Chen Change Loy Ziwei Liu MDE 40 177 0 28 Mar 2023
Learning to Grow Pretrained Models for Efficient Transformer Training Peihao Wang Rameswar Panda Lucas Torroba Hennigen P. Greengard Leonid Karlinsky Rogerio Feris David D. Cox Zhangyang Wang Yoon Kim 36 53 0 02 Mar 2023
A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity Hongkang Li M. Wang Sijia Liu Pin-Yu Chen ViT MLT 35 56 0 12 Feb 2023
A Survey on Efficient Training of Transformers Bohan Zhuang Jing Liu Zizheng Pan Haoyu He Yuetian Weng Chunhua Shen 25 47 0 02 Feb 2023
EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones Yulin Wang Yang Yue Rui Lu Tian-De Liu Zhaobai Zhong S. Song Gao Huang 34 28 0 17 Nov 2022
Progressive Transformation Learning for Leveraging Virtual Images in Training Yingzhe Shen Hyungtae Lee H. Kwon Shuvra S. Bhattacharyya 3DH 21 11 0 03 Nov 2022
Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities Brian Bartoldson B. Kailkhura Davis W. Blalock 31 47 0 13 Oct 2022
FocusFormer: Focusing on What We Need via Architecture Sampler Jing Liu Jianfei Cai Bohan Zhuang 29 7 0 23 Aug 2022
Transformers in Time-series Analysis: A Tutorial Sabeen Ahmed Ian E. Nielsen Aakash Tripathi Shamoon Siddiqui Ghulam Rasool R. Ramachandran AI4TS 28 142 0 28 Apr 2022
DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers Changlin Li Guangrun Wang Bing Wang Xiaodan Liang Zhihui Li Xiaojun Chang 30 9 0 21 Sep 2021
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 244 643 0 21 Apr 2021
StackRec: Efficient Training of Very Deep Sequential Recommender Models by Iterative Stacking Jiachun Wang Fajie Yuan Jian Chen Qingyao Wu Min Yang Yang Sun Guoxiao Zhang BDL 34 26 0 14 Dec 2020
On the Transformer Growth for Progressive BERT Training Xiaotao Gu Liyuan Liu Hongkun Yu Jing Li C. L. P. Chen Jiawei Han VLM 66 51 0 23 Oct 2020
L $^2$ -GCN: Layer-Wise and Learned Efficient Training of Graph Convolutional Networks Yuning You Tianlong Chen Zhangyang Wang Yang Shen GNN 101 82 0 30 Mar 2020
Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results Antti Tarvainen Harri Valpola OOD MoMe 261 1,275 0 06 Mar 2017
Neural Architecture Search with Reinforcement Learning Barret Zoph Quoc V. Le 269 5,326 0 05 Nov 2016