Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

23 January 2017

Papers citing "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer"

50 / 126 papers shown

Title
FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models Hao Kang Zichun Yu Chenyan Xiong MoE 20 0 0 26 May 2025
Co-AttenDWG: Co-Attentive Dimension-Wise Gating and Expert Fusion for Multi-Modal Offensive Content Detection Md. Mithun Hossain Md. Shakil Hossain Sudipto Chaki M. F. Mridha 70 0 0 25 May 2025
CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning Jinyuan Feng Chaopeng Wei Tenghai Qiu Tianyi Hu Zhiqiang Pu MoE 30 0 0 23 May 2025
Enhancing CTR Prediction with De-correlated Expert Networks Jiancheng Wang Mingjia Yin Junwei Pan Ximei Wang Hao Wang Enhong Chen 19 0 0 23 May 2025
Adaptive Semantic Token Communication for Transformer-based Edge Inference Alessio Devoto Jary Pomponi Mattia Merluzzi Paolo Di Lorenzo Simone Scardapane 61 0 0 23 May 2025
Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures Tuan Thai TrungTin Nguyen Dat Do Nhat Ho Christopher Drovandi 109 0 0 19 May 2025
Multi-modal Collaborative Optimization and Expansion Network for Event-assisted Single-eye Expression Recognition Runduo Han Xiuping Liu Shangxuan Yi Yi Zhang Hongchen Tan 62 0 0 17 May 2025
Chain-of-Model Learning for Language Model Kaitao Song Xiaohua Wang Xu Tan Huiqiang Jiang Chengruidong Zhang ... Xiaoqing Zheng Tao Qin Yuqing Yang Dongsheng Li Lili Qiu LRM AI4CE 70 0 0 17 May 2025
Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques Sanjay Surendranath Girija Shashank Kapoor Lakshit Arora Dipen Pradhan Aman Raj Ankit Shetgaonkar 71 0 0 05 May 2025
Backdoor Attacks Against Patch-based Mixture of Experts Cedric Chan Jona te Lintelo S. Picek AAML MoE 309 0 0 03 May 2025
Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing Piotr Piekos Róbert Csordás Jürgen Schmidhuber MoE VLM 150 2 0 01 May 2025
Unveiling the Hidden: Movie Genre and User Bias in Spoiler Detection Haokai Zhang Shengtao Zhang Zijian Cai Heng Wang Ruixuan Zhu Zinan Zeng Minnan Luo 81 0 0 24 Apr 2025
Hexcute: A Tile-based Programming Language with Automatic Layout and Task-Mapping Synthesis Xinsong Zhang Yaoyao Ding Yang Hu Gennady Pekhimenko 75 0 0 22 Apr 2025
MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core Dennis Liu Zijie Yan Xin Yao Tong Liu V. Korthikanti ... Jiajie Yao Chandler Zhou David Wu Xipeng Li J. Yang MoE 94 0 0 21 Apr 2025
Manipulating Multimodal Agents via Cross-Modal Prompt Injection Le Wang Zonghao Ying Tianyuan Zhang Siyuan Liang Shengshan Hu Mingchuan Zhang A. Liu Xianglong Liu AAML 74 1 0 19 Apr 2025
Dense Backpropagation Improves Training for Sparse Mixture-of-Experts Ashwinee Panda Vatsal Baherwani Zain Sarwar Benjamin Thérien Supriyo Chakraborty Tom Goldstein MoE 55 0 0 16 Apr 2025
Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design Mohan Zhang Pingzhi Li Jie Peng Mufan Qiu Tianlong Chen MoE 94 0 0 02 Apr 2025
Biologically Inspired Spiking Diffusion Model with Adaptive Lateral Selection Mechanism Linghao Feng Dongcheng Zhao Sicheng Shen Yi Zeng 81 0 0 31 Mar 2025
MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation Rongyu Zhang Menghang Dong Yuan Zhang Liang Heng Xiaowei Chi Gaole Dai Li Du Dan Wang Yuan Du MoE 99 1 0 26 Mar 2025
Reasoning Beyond Limits: Advances and Open Problems for LLMs M. Ferrag Norbert Tihanyi Merouane Debbah ELM OffRL LRM AI4CE 268 3 0 26 Mar 2025
BiPrompt-SAM: Enhancing Image Segmentation via Explicit Selection between Point and Text Prompts Suzhe Xu Jialin Peng Chengyuan Zhang VLM 76 0 0 25 Mar 2025
Resilient Sensor Fusion under Adverse Sensor Failures via Multi-Modal Expert Fusion Konyul Park Yecheol Kim Daehun Kim Jun-Won Choi 78 0 0 25 Mar 2025
Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization Minsu Kim Seongmin Hong RyeoWook Ko S. Choi Hunjong Lee Junsoo Kim Joo-Young Kim Jongse Park 67 0 0 24 Mar 2025
Mixture of Lookup Experts Shibo Jie Yehui Tang Kai Han Yongqian Li Duyu Tang Zhi-Hong Deng Yunhe Wang MoE 79 1 0 20 Mar 2025
A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications Siyuan Mu Sen Lin MoE 261 5 0 10 Mar 2025
RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs Zhongzhan Huang Guoming Ling Vincent S. Liang Yupei Lin Yandong Chen Shanshan Zhong Hefeng Wu LRM 94 5 0 08 Mar 2025
Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts Weigao Sun Disen Lan Tong Zhu Xiaoye Qu Yu Cheng MoE 144 2 0 07 Mar 2025
Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts Shwai He Weilin Cai Jiayi Huang Ang Li MoE 77 1 0 07 Mar 2025
The Society of HiveMind: Multi-Agent Optimization of Foundation Model Swarms to Unlock the Potential of Collective Intelligence Noah Mamie Susie Xi Rao LLMAG AI4CE 81 0 0 07 Mar 2025
TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecaster Kanghui Ning Zijie Pan Yu Liu Yushan Jiang Junxuan Zhang Kashif Rasul Anderson Schneider Lintao Ma Yuriy Nevmyvaka Dongjin Song AI4TS VLM 98 1 0 06 Mar 2025
Similarity-Distance-Magnitude Universal Verification Allen Schmaltz UQCV AAML 375 0 0 27 Feb 2025
Enhancing the Scalability and Applicability of Kohn-Sham Hamiltonians for Molecular Systems Yunyang Li Zaishuo Xia Lin Huang Xinran Wei Han Yang ... Zun Wang Chang-Shu Liu Jia Zhang Jia Zhang Mark B. Gerstein 102 2 0 26 Feb 2025
Filtered not Mixed: Stochastic Filtering-Based Online Gating for Mixture of Large Language Models Raeid Saqur Anastasis Kratsios Florian Krach Yannick Limmer Jacob-Junqi Tian John Willes Blanka Horvath Frank Rudzicz MoE 99 0 0 24 Feb 2025
Yes, Q-learning Helps Offline In-Context RL Denis Tarasov Alexander Nikulin Ilya Zisman Albina Klepach Andrei Polubarov Nikita Lyubaykin Alexander Derevyagin Igor Kiselev Vladislav Kurenkov OffRL OnRL 332 1 0 24 Feb 2025
Neural Attention Search Difan Deng Marius Lindauer 106 0 0 21 Feb 2025
Theory on Mixture-of-Experts in Continual Learning Hongbo Li Sen-Fon Lin Lingjie Duan Yingbin Liang Ness B. Shroff MoE MoMe CLL 203 16 0 20 Feb 2025
Stacking as Accelerated Gradient Descent Naman Agarwal Pranjal Awasthi Satyen Kale Eric Zhao ODL 82 2 0 20 Feb 2025
MoM: Linear Sequence Modeling with Mixture-of-Memories Jusen Du Weigao Sun Disen Lan Jiaxi Hu Yu Cheng KELM 93 3 0 19 Feb 2025
Forget the Data and Fine-Tuning! Just Fold the Network to Compress Dong Wang Haris Šikić Lothar Thiele O. Saukh 74 1 0 17 Feb 2025
Linear Mode Connectivity in Differentiable Tree Ensembles Ryuichi Kanoh M. Sugiyama 109 1 0 17 Feb 2025
Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling Xiao Li Zekai Zhang Xiang Li Siyi Chen Zhihui Zhu Peng Wang Qing Qu DiffM 88 0 0 09 Feb 2025
Skill Expansion and Composition in Parameter Space Tenglong Liu Junjie Li Yinan Zheng Haoyi Niu Yixing Lan Xin Xu Xianyuan Zhan 79 4 0 09 Feb 2025
Importance Sampling via Score-based Generative Models Heasung Kim Taekyun Lee Hyeji Kim Gustavo de Veciana MedIm DiffM 161 2 0 07 Feb 2025
MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generation Haibo Tong Zhaoyang Wang Zhe Chen Haonian Ji Shi Qiu ... Peng Xia Mingyu Ding Rafael Rafailov Chelsea Finn Huaxiu Yao EGVM VGen 134 3 0 03 Feb 2025
MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs Yuhang Zhou Giannis Karamanolakis Victor Soto Anna Rumshisky Mayank Kulkarni Furong Huang Wei Ai Jianhua Lu MoMe 120 2 0 03 Feb 2025
Multilingual State Space Models for Structured Question Answering in Indic Languages A. Vats Rahul Raja Mrinal Mathur Vinija Jain Aman Chadha 107 1 0 01 Feb 2025
Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization Zishun Yu Tengyu Xu Di Jin Karthik Abinav Sankararaman Yun He ... Eryk Helenowski Chen Zhu Sinong Wang Hao Ma Han Fang LRM 97 7 0 29 Jan 2025
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models Samira Abnar Harshay Shah Dan Busbridge Alaaeldin Mohamed Elnouby Ali J. Susskind Vimal Thilak MoE LRM 58 5 0 28 Jan 2025
Modality Interactive Mixture-of-Experts for Fake News Detection Yifan Liu Y. Liu Zehan Li Ruichen Yao Yang Zhang Dong Wang MoE 52 0 0 21 Jan 2025
Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models Zihan Qiu Zeyu Huang Jian Xu Kaiyue Wen Zhaoxiang Wang Rui Men Ivan Titov Dayiheng Liu Jingren Zhou Junyang Lin MoE 83 6 0 21 Jan 2025