MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation

15 April 2022

Papers citing "MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation"

24 / 24 papers shown

Title
Mixture of Routers Jia-Chen Zhang Yu-Jie Xiong Xi-He Qiu Chun-Ming Xia Fei Dai MoE 76 0 0 30 Mar 2025
Exploring the Benefit of Activation Sparsity in Pre-training Zhengyan Zhang Chaojun Xiao Qiujieli Qin Yankai Lin Zhiyuan Zeng Xu Han Zhiyuan Liu Ruobing Xie Maosong Sun Jie Zhou MoE 64 3 0 04 Oct 2024
CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information Yuxin Wang Minghua Ma Zekun Wang Jingchang Chen Huiming Fan Liping Shan Qing Yang Dongliang Xu Ming Liu Bing Qin 38 3 0 20 Sep 2024
LLAVADI: What Matters For Multimodal Large Language Models Distillation Shilin Xu Xiangtai Li Haobo Yuan Lu Qi Yunhai Tong Ming-Hsuan Yang 36 3 0 28 Jul 2024
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training Tong Zhu Xiaoye Qu Daize Dong Jiacheng Ruan Jingqi Tong Conghui He Yu Cheng MoE ALM 54 72 0 24 Jun 2024
Flexible and Adaptable Summarization via Expertise Separation Preslav Nakov Mingzhe Li Shen Gao Xin Cheng Qingqing Zhu Rui Yan Xin Gao Xiangliang Zhang MoE 44 3 0 08 Jun 2024
A Survey on Knowledge Distillation of Large Language Models Xiaohan Xu Ming Li Chongyang Tao Tao Shen Reynold Cheng Jinyang Li Can Xu Dacheng Tao Dinesh Manocha KELM VLM 44 103 0 20 Feb 2024
Model Compression and Efficient Inference for Large Language Models: A Survey Wenxiao Wang Wei Chen Yicong Luo Yongliu Long Zhengkai Lin Liye Zhang Binbin Lin Deng Cai Xiaofei He MQ 41 48 0 15 Feb 2024
On Parameter Estimation in Deviated Gaussian Mixture of Experts Huy Nguyen Khai Nguyen Nhat Ho 44 0 0 07 Feb 2024
Contrastive Learning and Mixture of Experts Enables Precise Vector Embeddings Logan Hallee Rohan Kapur Arjun Patel Jason P. Gleghorn Bohdan B. Khomtchouk MoE 22 3 0 28 Jan 2024
Language-driven All-in-one Adverse Weather Removal Hao Yang Liyuan Pan Yan Yang Wei Liang VLM KELM 34 18 0 03 Dec 2023
G-SPEED: General SParse Efficient Editing MoDel Haoke Zhang Yue Wang Juntao Li Xiabing Zhou Min Zhang SyDa KELM 30 1 0 16 Oct 2023
Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion Filip Szatkowski Eric Elmoznino Younesse Kaddar Simone Scardapane MoE 41 5 0 06 Oct 2023
Partition-and-Debias: Agnostic Biases Mitigation via A Mixture of Biases-Specific Experts Jiaxuan Li D. Vo Hideki Nakayama 26 3 0 19 Aug 2023
COMET: Learning Cardinality Constrained Mixture of Experts with Trees and Local Search Shibal Ibrahim Wenyu Chen Hussein Hazimeh Natalia Ponomareva Zhe Zhao Rahul Mazumder MoE 32 3 0 05 Jun 2023
Blockwise Parallel Transformer for Large Context Models Hao Liu Pieter Abbeel 49 11 0 30 May 2023
Lifting the Curse of Capacity Gap in Distilling Language Models Chen Zhang Yang Yang Jiahao Liu Jingang Wang Yunsen Xian Benyou Wang Dawei Song MoE 32 19 0 20 May 2023
Task-oriented Memory-efficient Pruning-Adapter Guorun Wang Jun Yang Yaoru Sun 42 4 0 26 Mar 2023
A Stability Analysis of Fine-Tuning a Pre-Trained Model Z. Fu Anthony Man-Cho So Nigel Collier 23 3 0 24 Jan 2023
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints Aran Komatsuzaki J. Puigcerver James Lee-Thorp Carlos Riquelme Ruiz Basil Mustafa Joshua Ainslie Yi Tay Mostafa Dehghani N. Houlsby MoMe MoE 29 109 0 09 Dec 2022
Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling Haw-Shiuan Chang Ruei-Yao Sun Kathryn Ricci Andrew McCallum 43 15 0 10 Oct 2022
Less is More: Task-aware Layer-wise Distillation for Language Model Compression Chen Liang Simiao Zuo Qingru Zhang Pengcheng He Weizhu Chen Tuo Zhao VLM 50 68 0 04 Oct 2022
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing Canwen Xu Wangchunshu Zhou Tao Ge Furu Wei Ming Zhou 231 198 0 07 Feb 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 299 6,996 0 20 Apr 2018