A Provably Effective Method for Pruning Experts in Fine-tuned Sparse
Mixture-of-Experts

A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts

26 May 2024

Mohammed Nowaz Rabbani Chowdhury

K. E. Maghraoui

Christopher Carothers

Papers citing "A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts"

9 / 9 papers shown

Title
QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration HamidReza Imani Jiaxin Peng Peiman Mohseni Abdolah Amirany Tarek A. El-Ghazawi MoE 26 0 0 10 May 2025
MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router Yanyue Xie Zhi Zhang Ding Zhou Cong Xie Ziang Song Xin Liu Yanzhi Wang Xue Lin An Xu LLMAG 38 3 0 15 Oct 2024
Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts Zeliang Zhang Xiaodong Liu Hao Cheng Chenliang Xu Jianfeng Gao MoE 32 9 0 12 Jul 2024
How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? Hongkang Li Meng Wang Songtao Lu Xiaodong Cui Pin-Yu Chen MLT 40 14 0 23 Feb 2024
On the Convergence and Sample Complexity Analysis of Deep Q-Networks with $ε$ -Greedy Exploration Shuai Zhang Hongkang Li Meng Wang Miao Liu Pin-Yu Chen Songtao Lu Sijia Liu K. Murugesan Subhajit Chaudhury 32 19 0 24 Oct 2023
Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models A. Jaiswal Shiwei Liu Tianlong Chen Ying Ding Zhangyang Wang VLM 32 22 0 18 Jun 2023
Mixture-of-Experts with Expert Choice Routing Yan-Quan Zhou Tao Lei Han-Chu Liu Nan Du Yanping Huang Vincent Zhao Andrew M. Dai Zhifeng Chen Quoc V. Le James Laudon MoE 160 327 0 18 Feb 2022
The Lottery Ticket Hypothesis for Pre-trained BERT Networks Tianlong Chen Jonathan Frankle Shiyu Chang Sijia Liu Yang Zhang Zhangyang Wang Michael Carbin 148 345 0 23 Jul 2020
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 296 39,194 0 01 Sep 2014