Sparse Modular Activation for Efficient Sequence Modeling

Sparse Modular Activation for Efficient Sequence Modeling

19 June 2023

Yang Liu

Shuohang Wang

Chengxiang Zhai

Papers citing "Sparse Modular Activation for Efficient Sequence Modeling"

11 / 11 papers shown

Title
Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions Zhihao He Hang Yu Zi Gong Shizhan Liu J. Li Weiyao Lin VLM 38 1 0 09 Oct 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 74 56 0 11 Jun 2024
Investigating Recurrent Transformers with Dynamic Halt Jishnu Ray Chowdhury Cornelia Caragea 39 1 0 01 Feb 2024
Language Model Pre-Training with Sparse Latent Typing Liliang Ren Zixuan Zhang H. Wang Clare R. Voss Chengxiang Zhai Heng Ji 48 3 0 23 Oct 2022
Why neural networks find simple solutions: the many regularizers of geometric complexity Benoit Dherin Michael Munn M. Rosca David Barrett 57 31 0 27 Sep 2022
Liquid Structural State-Space Models Ramin Hasani Mathias Lechner Tsun-Hsuan Wang Makram Chahine Alexander Amini Daniela Rus AI4TS 107 96 0 26 Sep 2022
Transkimmer: Transformer Learns to Layer-wise Skim Yue Guan Zhengyi Li Jingwen Leng Zhouhan Lin Minyi Guo 80 38 0 15 May 2022
Transformer Quality in Linear Time Weizhe Hua Zihang Dai Hanxiao Liu Quoc V. Le 78 222 0 21 Feb 2022
Mixture-of-Experts with Expert Choice Routing Yan-Quan Zhou Tao Lei Han-Chu Liu Nan Du Yanping Huang Vincent Zhao Andrew M. Dai Zhifeng Chen Quoc V. Le James Laudon MoE 160 329 0 18 Feb 2022
Primer: Searching for Efficient Transformers for Language Modeling David R. So Wojciech Mañke Hanxiao Liu Zihang Dai Noam M. Shazeer Quoc V. Le VLM 91 152 0 17 Sep 2021
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 252 580 0 12 Mar 2020