GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

30 June 2020

Papers citing "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding"

50 / 260 papers shown

Title
CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts Zhenpeng Su Xing Wu Zijia Lin Yizhe Xiong Minxuan Lv Guangyuan Ma Hui Chen Songlin Hu Guiguang Ding MoE 29 3 0 21 Oct 2024
MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts R. Teo Tan M. Nguyen MoE 33 3 0 18 Oct 2024
EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference Yulei Qian Fengcun Li Xiangyang Ji Xiaoyu Zhao Jianchao Tan Kaipeng Zhang Xunliang Cai MoE 79 3 0 16 Oct 2024
MoH: Multi-Head Attention as Mixture-of-Head Attention Peng Jin Bo Zhu Li Yuan Shuicheng Yan MoE 31 13 0 15 Oct 2024
Quadratic Gating Functions in Mixture of Experts: A Statistical Insight Pedram Akbarian Huy Le Nguyen Xing Han Nhat Ho MoE 42 0 0 15 Oct 2024
GaVaMoE: Gaussian-Variational Gated Mixture of Experts for Explainable Recommendation Fei Tang Yongliang Shen Hang Zhang Zeqi Tan Wenqi Zhang Guiyang Hou Kaitao Song Weiming Lu Yueting Zhuang 50 0 0 15 Oct 2024
Ada-K Routing: Boosting the Efficiency of MoE-based LLMs Tongtian Yue Longteng Guo Jie Cheng Xuange Gao Jiaheng Liu MoE 39 0 0 14 Oct 2024
Scalable Multi-Domain Adaptation of Language Models using Modular Experts Peter Schafhalter Shun Liao Yanqi Zhou Chih-Kuan Yeh Arun Kandoor James Laudon MoE 29 1 0 14 Oct 2024
Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts Guorui Zheng Xidong Wang Juhao Liang Nuo Chen Yuping Zheng Benyou Wang MoE 35 5 0 14 Oct 2024
GETS: Ensemble Temperature Scaling for Calibration in Graph Neural Networks Dingyi Zhuang Chonghe Jiang Yunhan Zheng Shenhao Wang Jinhua Zhao UQCV 39 0 0 12 Oct 2024
Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMs Ruijia Niu D. Wu Rose Yu Yi Ma 33 1 0 09 Oct 2024
Aria: An Open Multimodal Native Mixture-of-Experts Model Dongxu Li Yudong Liu Haoning Wu Yue Wang Zhiqi Shen ... Lihuan Zhang Hanshu Yan Guoyin Wang Bei Chen Junnan Li MoE 51 48 0 08 Oct 2024
X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale Haoran Xu Kenton W. Murray Philipp Koehn Hieu T. Hoang Akiko Eriguchi Huda Khayrallah 37 8 0 04 Oct 2024
No Need to Talk: Asynchronous Mixture of Language Models Anastasiia Filippova Angelos Katharopoulos David Grangier Ronan Collobert MoE 44 0 0 04 Oct 2024
Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL Ghada Sokar J. Obando-Ceron Rameswar Panda Hugo Larochelle Pablo Samuel Castro MoE 142 2 0 02 Oct 2024
EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing Haotian Sun Tao Lei Bowen Zhang Yanghao Li Haoshuo Huang Ruoming Pang Bo Dai Nan Du DiffM MoE 81 5 0 02 Oct 2024
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts Xiaoming Shi Shiyu Wang Yuqi Nie Dianqi Li Zhou Ye Qingsong Wen Ming Jin AI4TS 46 28 0 24 Sep 2024
Customizing Language Models with Instance-wise LoRA for Sequential Recommendation Xiaoyu Kong Jiancan Wu An Zhang Leheng Sheng Hui Lin Xiang Wang Xiangnan He AI4TS 61 7 0 19 Aug 2024
Layerwise Recurrent Router for Mixture-of-Experts Zihan Qiu Zeyu Huang Shuang Cheng Yizhi Zhou Zili Wang Ivan Titov Jie Fu MoE 81 2 0 13 Aug 2024
Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustness Xiaojing Fan Chunliang Tao AAML 39 28 0 08 Aug 2024
Understanding the Performance and Estimating the Cost of LLM Fine-Tuning Yuchen Xia Jiho Kim Yuhan Chen Haojie Ye Souvik Kundu Cong Hao Nishil Talati MoE 37 22 0 08 Aug 2024
MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts Xi Lin Akshat Shrivastava Liang Luo Srinivasan Iyer Mike Lewis Gargi Gosh Luke Zettlemoyer Armen Aghajanyan MoE 43 20 0 31 Jul 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 42 43 0 09 Jul 2024
FRED: Flexible REduction-Distribution Interconnect and Communication Implementation for Wafer-Scale Distributed Training of DNN Models Saeed Rashidi William Won Sudarshan Srinivasan Puneet Gupta Tushar Krishna 30 0 0 28 Jun 2024
Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model Longrong Yang Dong Shen Chaoxiang Cai Fan Yang Size Li Di Zhang Xi Li MoE 56 2 0 28 Jun 2024
$$\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts$ $\texttt{MoE-RBench}$ : Towards Building Reliable Language Models with Sparse Mixture-of-Experts Guanjie Chen Xinyu Zhao Tianlong Chen Yu Cheng MoE 76 5 0 17 Jun 2024
Scorch: A Library for Sparse Deep Learning Bobby Yan Alexander J. Root Trevor Gale David Broman Fredrik Kjolstad 33 0 0 27 May 2024
Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling Cristian Rodriguez-Opazo Ehsan Abbasnejad Damien Teney Edison Marrese-Taylor Hamed Damirchi Anton Van Den Hengel VLM 40 1 0 27 May 2024
Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models Yongxin Guo Zhenglin Cheng Xiaoying Tang Tao R. Lin Tao Lin MoE 66 7 0 23 May 2024
Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts Yunxin Li Shenyuan Jiang Baotian Hu Longyue Wang Wanqi Zhong Wenhan Luo Lin Ma Min-Ling Zhang MoE 46 30 0 18 May 2024
A Mixture of Experts Approach to 3D Human Motion Prediction Edmund Shieh Joshua Lee Franco Kang Min Bae Tej Lalvani 32 1 0 09 May 2024
Multi-Head Mixture-of-Experts Xun Wu Shaohan Huang Wenhui Wang Furu Wei MoE 39 12 0 23 Apr 2024
From Matching to Generation: A Survey on Generative Information Retrieval Xiaoxi Li Jiajie Jin Yujia Zhou Yuyao Zhang Peitian Zhang Yutao Zhu Zhicheng Dou 3DV 84 46 0 23 Apr 2024
Model Callers for Transforming Predictive and Generative AI Applications Mukesh Dalal 26 0 0 17 Apr 2024
Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies Benjue Weng LM&MA 46 8 0 13 Apr 2024
Mixture of Low-rank Experts for Transferable AI-Generated Image Detection Zihan Liu Hanyi Wang Yaoyu Kang Shilin Wang MoE 41 12 0 07 Apr 2024
Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights Moein Heidari Reza Azad Sina Ghorbani Kolahi René Arimond Leon Niggemeier ... Afshin Bozorgpour Ehsan Khodapanah Aghdam A. Kazerouni I. Hacihaliloglu Dorit Merhof 51 7 0 28 Mar 2024
A Codesign of Scheduling and Parallelization for Large Model Training in Heterogeneous Clusters Chunyu Xue Weihao Cui Han Zhao Quan Chen Shulai Zhang Peng Yang Jing Yang Shaobo Li Minyi Guo 45 2 0 24 Mar 2024
DiPaCo: Distributed Path Composition Arthur Douillard Qixuang Feng Andrei A. Rusu A. Kuncoro Yani Donchev Rachita Chhaparia Ionel Gog MarcÁurelio Ranzato Jiajun Shen Arthur Szlam MoE 48 2 0 15 Mar 2024
Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models Xudong Lu Qi Liu Yuhui Xu Aojun Zhou Siyuan Huang Bo-Wen Zhang Junchi Yan Hongsheng Li MoE 32 26 0 22 Feb 2024
LLMBind: A Unified Modality-Task Integration Framework Bin Zhu Munan Ning Peng Jin Bin Lin Jinfa Huang ... Junwu Zhang Zhenyu Tang Mingjun Pan Xing Zhou Li-ming Yuan MLLM 40 6 0 22 Feb 2024
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models Chris Liu Renrui Zhang Longtian Qiu Siyuan Huang Weifeng Lin ... Hao Shao Pan Lu Hongsheng Li Yu Qiao Peng Gao MLLM 130 109 0 08 Feb 2024
ReLU $^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs Zhengyan Zhang Yixin Song Guanghui Yu Xu Han Yankai Lin Chaojun Xiao Chenyang Song Zhiyuan Liu Zeyu Mi Maosong Sun 22 31 0 06 Feb 2024
Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts Anastasis Kratsios Haitz Sáez de Ocáriz Borde Takashi Furuya Marc T. Law MoE 41 1 0 05 Feb 2024
LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs Shaoxiang Chen Zequn Jie Lin Ma MoE 45 47 0 29 Jan 2024
LocMoE: A Low-Overhead MoE for Large Language Model Training Jing Li Zhijie Sun Xuan He Li Zeng Yi Lin Entong Li Binfan Zheng Rongqian Zhao Xin Chen MoE 30 11 0 25 Jan 2024
HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis Shiwei Zhang Lansong Diao Chuan Wu Zongyan Cao Siyu Wang Wei Lin 43 12 0 11 Jan 2024
An Empirical Study of Scaling Law for OCR Miao Rang Zhenni Bi Chuanjian Liu Yunhe Wang Kai Han 41 6 0 29 Dec 2023
Gene-MOE: A sparsely gated prognosis and classification framework exploiting pan-cancer genomic information Xiangyu Meng Xue Li Qing Yang Huanhuan Dai Lian Qiao Hongzhen Ding Long Hao Xun Wang 11 0 0 29 Nov 2023
Conditional Prompt Tuning for Multimodal Fusion Ruixia Jiang Lingbo Liu Changwen Chen 25 0 0 28 Nov 2023