AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient
Pre-trained Language Models

AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models

29 July 2021

Lifeng Shang

Xin Jiang

Qun Liu

Papers citing "AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models"

12 / 12 papers shown

Title
Merino: Entropy-driven Design for Generative Language Models on IoT Devices Youpeng Zhao Ming Lin Huadong Tang Qiang Wu Jun Wang 86 0 0 28 Jan 2025
YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUs Cyrus Zhou Zack Hassman Ruize Xu Dhirpal Shah Vaughn Richard Yanjing Li 34 1 0 01 Oct 2023
Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review M. Wong Shangxin Guo Ching Nam Hang Siu-Wai Ho C. Tan 47 78 0 04 Jul 2023
Gradient-Free Structured Pruning with Unlabeled Data Azade Nova H. Dai Dale Schuurmans SyDa 40 20 0 07 Mar 2023
MUX-PLMs: Data Multiplexing for High-throughput Language Models Vishvak Murahari Ameet Deshpande Carlos E. Jimenez Izhak Shafran Mingqiu Wang Yuan Cao Karthik Narasimhan MoE 31 5 0 24 Feb 2023
Gradient-based Intra-attention Pruning on Pre-trained Language Models Ziqing Yang Yiming Cui Xin Yao Shijin Wang VLM 42 8 0 15 Dec 2022
Structured Pruning Learns Compact and Accurate Models Mengzhou Xia Zexuan Zhong Danqi Chen VLM 21 180 0 01 Apr 2022
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models Dongkuan Xu Subhabrata Mukherjee Xiaodong Liu Debadeepta Dey Wenhui Wang Xiang Zhang Ahmed Hassan Awadallah Jianfeng Gao 33 4 0 29 Jan 2022
Towards Efficient Post-training Quantization of Pre-trained Language Models Haoli Bai Lu Hou Lifeng Shang Xin Jiang Irwin King M. Lyu MQ 82 47 0 30 Sep 2021
Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation Cheng Chen Yichun Yin Lifeng Shang Zhi Wang Xin Jiang Xiao Chen Qun Liu FedML 28 7 0 24 Apr 2021
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT Sheng Shen Zhen Dong Jiayu Ye Linjian Ma Z. Yao A. Gholami Michael W. Mahoney Kurt Keutzer MQ 236 578 0 12 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 304 7,005 0 20 Apr 2018