Go Wider Instead of Deeper

25 July 2021

Fuzhao Xue

Ziji Shi

Futao Wei

Yuxuan Lou

Yong Liu

Yang You

ViT

MoE

ArXiv PDF HTML

Papers citing "Go Wider Instead of Deeper"

49 / 49 papers shown

Title
3D Gaussian Splatting Data Compression with Mixture of Priors Lei Liu Zhenghao Chen Dong Xu 3DGS 169 0 0 06 May 2025
Mixture of Routers Jia-Chen Zhang Yu-Jie Xiong Xi-He Qiu Chun-Ming Xia Fei Dai MoE 71 0 0 30 Mar 2025
ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts Xumeng Han Longhui Wei Zhiyang Dou Zipeng Wang Chenhui Qiang Xin He Yingfei Sun Zhenjun Han Qi Tian MoE 45 3 0 21 Oct 2024
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models Cong Guo Feng Cheng Zhixu Du James Kiessling Jonathan Ku ... Qilin Zheng Guanglei Zhou Hai Li-Wei Li Yiran Chen 31 7 0 08 Oct 2024
All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation Xu Zhang Peiyao Guo Ming-Tse Lu Zhan Ma 43 2 0 29 Sep 2024
Layerwise Recurrent Router for Mixture-of-Experts Zihan Qiu Zeyu Huang Shuang Cheng Yizhi Zhou Zili Wang Ivan Titov Jie Fu MoE 81 2 0 13 Aug 2024
Mixture of Nested Experts: Adaptive Processing of Visual Tokens Gagan Jain Nidhi Hegde Aditya Kusupati Arsha Nagrani Shyamal Buch Prateek Jain Anurag Arnab Sujoy Paul MoE 45 7 0 29 Jul 2024
SACNet: A Spatially Adaptive Convolution Network for 2D Multi-organ Medical Segmentation Lin Zhang Wenbo Gao Jie Yi Yunyun Yang 46 0 0 14 Jul 2024
Powering In-Database Dynamic Model Slicing for Structured Data Analytics Lingze Zeng Naili Xing Shaofeng Cai Gang Chen Bengchin Ooi Jian Pei Yuncheng Wu 23 1 0 01 May 2024
Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts Weilin Cai Juyong Jiang Le Qin Junwei Cui Sunghun Kim Jiayi Huang 53 7 0 07 Apr 2024
PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition Chenhongyi Yang Zehui Chen Miguel Espinosa Linus Ericsson Zhenyu Wang Jiaming Liu Elliot J. Crowley Mamba 39 88 0 26 Mar 2024
DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling Shanghaoran Quan MoE OffRL 52 9 0 02 Mar 2024
Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization James Oldfield Markos Georgopoulos Grigorios G. Chrysos Christos Tzelepis Yannis Panagakis M. Nicolaou Jiankang Deng Ioannis Patras MoE 45 8 0 19 Feb 2024
OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models Fuzhao Xue Zian Zheng Yao Fu Jinjie Ni Zangwei Zheng Wangchunshu Zhou Yang You MoE 30 87 0 29 Jan 2024
PanGu- $π$ : Enhancing Language Model Architectures via Nonlinearity Compensation Yunhe Wang Hanting Chen Yehui Tang Tianyu Guo Kai Han ... Qinghua Xu Qun Liu Jun Yao Chao Xu Dacheng Tao 67 15 0 27 Dec 2023
A Survey of Reasoning with Foundation Models Jiankai Sun Chuanyang Zheng E. Xie Zhengying Liu Ruihang Chu ... Xipeng Qiu Yi-Chen Guo Hui Xiong Qun Liu Zhenguo Li ReLM LRM AI4CE 27 76 0 17 Dec 2023
SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models Zhixu Du Shiyu Li Yuhao Wu Xiangyu Jiang Jingwei Sun Qilin Zheng Yongkai Wu Ang Li Hai Helen Li Yiran Chen MoE 31 12 0 29 Oct 2023
Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models Weize Chen Xiaoyue Xu Xu Han Yankai Lin Ruobing Xie Zhiyuan Liu Maosong Sun Jie Zhou 34 0 0 19 Oct 2023
RigNet++: Semantic Assisted Repetitive Image Guided Network for Depth Completion Zhiqiang Yan Xiang Li Le Hui Zhenyu Zhang Jun Yu Li Jian Yang VLM 3DV 23 4 0 01 Sep 2023
Robust Mixture-of-Expert Training for Convolutional Neural Networks Yihua Zhang Ruisi Cai Tianlong Chen Guanhua Zhang Huan Zhang Pin-Yu Chen Shiyu Chang Zhangyang Wang Sijia Liu MoE AAML OOD 34 16 0 19 Aug 2023
Experts Weights Averaging: A New General Training Scheme for Vision Transformers Yongqian Huang Peng Ye Xiaoshui Huang Sheng Li Tao Chen Tong He Wanli Ouyang MoMe 28 8 0 11 Aug 2023
Efficient ResNets: Residual Network Design Aditya Thakur Harish Chauhan Nikunj Gupta 21 0 0 21 Jun 2023
ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning Xiao Xu Bei Li Chenfei Wu Shao-Yen Tseng Anahita Bhiwandiwalla Shachar Rosenman Vasudev Lal Wanxiang Che Nan Duan AIFin VLM 34 2 0 31 May 2023
To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis Fuzhao Xue Yao Fu Wangchunshu Zhou Zangwei Zheng Yang You 83 77 0 22 May 2023
Transforming Visual Scene Graphs to Image Captions Xu Yang Jiawei Peng Zihua Wang Haiyang Xu Qinghao Ye Chenliang Li Mingshi Yan Feisi Huang Zhangzikang Li Yu Zhang 49 19 0 03 May 2023
Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited Dataset Thanh-Dung Le P. Jouvet R. Noumeir MoE MedIm 72 5 0 22 Mar 2023
A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training Siddharth Singh Olatunji Ruwase A. A. Awan Samyam Rajbhandari Yuxiong He A. Bhatele MoE 40 30 0 11 Mar 2023
Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference Haiyang Huang Newsha Ardalani Anna Y. Sun Liu Ke Hsien-Hsin S. Lee Anjali Sridhar Shruti Bhosale Carole-Jean Wu Benjamin C. Lee MoE 65 23 0 10 Mar 2023
TAP: Accelerating Large-Scale DNN Training Through Tensor Automatic Parallelisation Ziji Shi Le Jiang Ang Wang Jie Zhang Xianyan Jia Yong Li Chencan Wu Jialin Li Wei Lin GNN 44 2 0 01 Feb 2023
Out of Distribution Performance of State of Art Vision Model Salman Rahman W. Lee 37 2 0 25 Jan 2023
On the Adversarial Robustness of Mixture of Experts J. Puigcerver Rodolphe Jenatton C. Riquelme Pranjal Awasthi Srinadh Bhojanapalli OOD AAML MoE 37 18 0 19 Oct 2022
Wide Attention Is The Way Forward For Transformers? Jason Brown Yiren Zhao Ilia Shumailov Robert D. Mullins 21 7 0 02 Oct 2022
Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition Ye Bai Jie Li W. Han Hao Ni Kaituo Xu Zhuo Zhang Cheng Yi Xiaorui Wang MoE 21 1 0 17 Sep 2022
MoEC: Mixture of Expert Clusters Yuan Xie Shaohan Huang Tianyu Chen Furu Wei MoE 40 11 0 19 Jul 2022
Training Transformers Together Alexander Borzunov Max Ryabinin Tim Dettmers Quentin Lhoest Lucile Saulnier Michael Diskin Yacine Jernite Thomas Wolf ViT 28 8 0 07 Jul 2022
Can CNNs Be More Robust Than Transformers? Zeyu Wang Yutong Bai Yuyin Zhou Cihang Xie UQCV OOD 22 46 0 07 Jun 2022
A Study on Transformer Configuration and Training Objective Fuzhao Xue Jianghai Chen Aixin Sun Xiaozhe Ren Zangwei Zheng Xiaoxin He Yongming Chen Xin Jiang Yang You 33 7 0 21 May 2022
Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability Svetlana Pavlitska Christian Hubschneider Lukas Struppek J. Marius Zöllner MoE 24 11 0 22 Apr 2022
Are Vision Transformers Robust to Spurious Correlations? Soumya Suvra Ghosal Yifei Ming Yixuan Li ViT 25 28 0 17 Mar 2022
One Student Knows All Experts Know: From Sparse to Dense Fuzhao Xue Xiaoxin He Xiaozhe Ren Yuxuan Lou Yang You MoMe MoE 27 20 0 26 Jan 2022
Are Transformers More Robust Than CNNs? Yutong Bai Jieru Mei Alan Yuille Cihang Xie ViT AAML 192 257 0 10 Nov 2021
Large-Scale Deep Learning Optimizations: A Comprehensive Survey Xiaoxin He Fuzhao Xue Xiaozhe Ren Yang You 27 14 0 01 Nov 2021
Sparse MoEs meet Efficient Ensembles J. Allingham F. Wenzel Zelda E. Mariet Basil Mustafa J. Puigcerver ... Balaji Lakshminarayanan Jasper Snoek Dustin Tran Carlos Riquelme Ruiz Rodolphe Jenatton MoE 46 21 0 07 Oct 2021
Cross-token Modeling with Conditional Computation Yuxuan Lou Fuzhao Xue Zangwei Zheng Yang You MoE 38 19 0 05 Sep 2021
Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters Chen Sun Shenggui Li Jinyue Wang Jun Yu 54 47 0 08 Aug 2021
Concurrent Adversarial Learning for Large-Batch Training Yong Liu Xiangning Chen Minhao Cheng Cho-Jui Hsieh Yang You ODL 28 13 0 01 Jun 2021
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing Canwen Xu Wangchunshu Zhou Tao Ge Furu Wei Ming Zhou 221 197 0 07 Feb 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018