Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models

2 March 2022

Papers citing "Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models"

22 / 22 papers shown

Title
Reproducibility Study of "Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents" Pedro M. P. Curvo Mara Dragomir Salvador Torpes Mohammadmahdi Rahimi LLMAG 16 0 0 14 May 2025
Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations Zican Dong Han Peng Peiyu Liu Wayne Xin Zhao Dong Wu Feng Xiao Zhongqi Wang MoE 33 0 0 09 Apr 2025
EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Models Yinan Liang Zhi Wang Xiuwei Xu Jie Zhou Jiwen Lu VLM LRM 48 0 0 19 Mar 2025
eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference Suraiya Tairin Shohaib Mahmud Haiying Shen Anand Iyer MoE 158 0 0 10 Mar 2025
Sample Selection via Contrastive Fragmentation for Noisy Label Regression C. Kim Sangwoo Moon Jihwan Moon Dongyeon Woo Gunhee Kim NoLa 57 0 0 25 Feb 2025
Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation Yu-Liang Zhan Zhong-Yi Lu Hao Sun Ze-Feng Gao 30 0 0 10 Nov 2024
MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router Yanyue Xie Zhi Zhang Ding Zhou Cong Xie Ziang Song Xin Liu Yanzhi Wang Xue Lin An Xu LLMAG 38 3 0 15 Oct 2024
MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfer Minghao Zhu Zhengpu Wang Mengxian Hu Ronghao Dang Xiao Lin Xun Zhou Chengju Liu Qijun Chen 37 1 0 14 Oct 2024
Exploring the Benefit of Activation Sparsity in Pre-training Zhengyan Zhang Chaojun Xiao Qiujieli Qin Yankai Lin Zhiyuan Zeng Xu Han Zhiyuan Liu Ruobing Xie Maosong Sun Jie Zhou MoE 64 3 0 04 Oct 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu-Xiang Wang 46 83 0 22 Apr 2024
Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods Yuji Cao Huan Zhao Yuheng Cheng Ting Shu Guolong Liu Gaoqi Liang Junhua Zhao Yun Li LLMAG KELM OffRL LM&Ro 35 49 0 30 Mar 2024
Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization James Oldfield Markos Georgopoulos Grigorios G. Chrysos Christos Tzelepis Yannis Panagakis M. Nicolaou Jiankang Deng Ioannis Patras MoE 42 8 0 19 Feb 2024
Model Compression and Efficient Inference for Large Language Models: A Survey Wenxiao Wang Wei Chen Yicong Luo Yongliu Long Zhengkai Lin Liye Zhang Binbin Lin Deng Cai Xiaofei He MQ 41 47 0 15 Feb 2024
Merging Experts into One: Improving Computational Efficiency of Mixture of Experts Shwai He Run-Ze Fan Liang Ding Li Shen Tianyi Zhou Dacheng Tao MoE MoMe 32 14 0 15 Oct 2023
Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer Boan Liu Liang Ding Li Shen Keqin Peng Yu Cao Dazhao Cheng Dacheng Tao MoE 36 7 0 15 Oct 2023
Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy Pingzhi Li Zhenyu (Allen) Zhang Prateek Yadav Yi-Lin Sung Yu Cheng Mohit Bansal Tianlong Chen MoMe 26 33 0 02 Oct 2023
Domain-Agnostic Neural Architecture for Class Incremental Continual Learning in Document Processing Platform Mateusz Wójcik Witold Ko'sciukiewicz Mateusz Baran Tomasz Kajdanowicz Adam Gonczarek CLL 19 1 0 11 Jul 2023
Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity Da Xu Maha Elbayad Kenton W. Murray Jean Maillard Vedanuj Goswami MoE 47 3 0 03 May 2023
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference Sneha Kudugunta Yanping Huang Ankur Bapna M. Krikun Dmitry Lepikhin Minh-Thang Luong Orhan Firat MoE 119 106 0 24 Sep 2021
Scalable and Efficient MoE Training for Multitask Multilingual Models Young Jin Kim A. A. Awan Alexandre Muzio Andres Felipe Cruz Salinas Liyang Lu Amr Hendy Samyam Rajbhandari Yuxiong He Hany Awadalla MoE 96 84 0 22 Sep 2021
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning Runxin Xu Fuli Luo Zhiyuan Zhang Chuanqi Tan Baobao Chang Songfang Huang Fei Huang LRM 145 178 0 13 Sep 2021
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018