Efficient Content-Based Sparse Attention with Routing Transformers

12 March 2020

Papers citing "Efficient Content-Based Sparse Attention with Routing Transformers"

50 / 116 papers shown

Title
Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing Piotr Piekos Róbert Csordás Jürgen Schmidhuber MoE VLM 96 1 0 01 May 2025
Cognitive Memory in Large Language Models Lianlei Shan Shixian Luo Zezhou Zhu Yu Yuan Yong Wu LLMAG KELM 152 1 0 03 Apr 2025
Attention Condensation via Sparsity Induced Regularized Training Eli Sason Darya Frolova Boris Nazarov Felix Goldberd 175 0 0 03 Mar 2025
Simplified and Generalized Masked Diffusion for Discrete Data Jiaxin Shi Kehang Han Z. Wang Arnaud Doucet Michalis K. Titsias DiffM 76 62 0 17 Jan 2025
Likelihood Training of Cascaded Diffusion Models via Hierarchical Volume-preserving Maps Henry Li Ronen Basri Y. Kluger DiffM 54 2 0 13 Jan 2025
Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Pingyi Chen Zhongyi Shui Chenglu Zhu Lin Yang MedIm 32 4 0 18 Oct 2024
Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity Mutian He Philip N. Garner 80 0 0 09 Oct 2024
Sampling Foundational Transformer: A Theoretical Perspective Viet Anh Nguyen Minh Lenhat Khoa Nguyen Duong Duc Hieu Dao Huu Hung Truong Son-Hy 44 0 0 11 Aug 2024
MambaGesture: Enhancing Co-Speech Gesture Generation with Mamba and Disentangled Multi-Modality Fusion Chencan Fu Yabiao Wang Jiangning Zhang Zhengkai Jiang Xiaofeng Mao Jiafu Wu Weijian Cao Chengjie Wang Yanhao Ge Yong Liu Mamba 35 2 0 29 Jul 2024
Let the Code LLM Edit Itself When You Edit the Code Zhenyu He Jun Zhang Shengjie Luo Jingjing Xu Z. Zhang Di He KELM 31 0 0 03 Jul 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 66 55 0 11 Jun 2024
Wavelet-Based Image Tokenizer for Vision Transformers Zhenhai Zhu Radu Soricut ViT 42 3 0 28 May 2024
Reducing the Barriers to Entry for Foundation Model Training Paolo Faraboschi Ellis Giles Justin Hotard Konstanty Owczarek Andrew Wheeler 27 4 0 12 Apr 2024
On the Theoretical Expressive Power and the Design Space of Higher-Order Graph Transformers Cai Zhou Rose Yu Yusu Wang 32 7 0 04 Apr 2024
Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling Mahdi Karami Ali Ghodsi VLM 40 6 0 28 Feb 2024
Multimodal Transformer With a Low-Computational-Cost Guarantee Sungjin Park Edward Choi 46 1 0 23 Feb 2024
Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalness Sicheng Yang Zunnan Xu Haiwei Xue Yongkang Cheng Shaoli Huang Mingming Gong Zhiyong Wu DiffM VGen 27 11 0 07 Jan 2024
Efficiency-oriented approaches for self-supervised speech representation learning Luis Lugo Valentin Vielzeuf SSL 23 1 0 18 Dec 2023
Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Chenglu Zhu Jiatong Cai Sunyi Zheng Lin Yang VLM 27 4 0 21 Nov 2023
Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attention Ziwei He Jian Yuan Le Zhou Jingwen Leng Bo Jiang 27 0 0 13 Nov 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 26 15 0 28 Sep 2023
UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons Sicheng Yang Z. Wang Zhiyong Wu Minglei Li Zhensong Zhang ... Lei Hao Songcen Xu Xiaofei Wu Changpeng Yang Zonghong Dai DiffM 39 14 0 13 Sep 2023
Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech Hyungchan Yoon Changhwan Kim Eunwoo Song Hyun-Wook Yoon Hong-Goo Kang 19 1 0 28 Aug 2023
Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers Tobias Christian Nauen Sebastián M. Palacio Federico Raue Andreas Dengel 39 3 0 18 Aug 2023
RecycleGPT: An Autoregressive Language Model with Recyclable Module Yu Jiang Qiaozhi He Xiaomin Zhuang Zhihua Wu Kunpeng Wang Wenlai Zhao Guangwen Yang KELM 23 3 0 07 Aug 2023
Attention over pre-trained Sentence Embeddings for Long Document Classification Amine Abdaoui Sourav Dutta 6 1 0 18 Jul 2023
Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis Zhiyu Jin Xuli Shen Bin Li Xiangyang Xue 24 36 0 14 Jun 2023
A Critical Evaluation of Evaluations for Long-form Question Answering Fangyuan Xu Yixiao Song Mohit Iyyer Eunsol Choi ELM 37 94 0 29 May 2023
Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator Ziwei He Meng-Da Yang Minwei Feng Jingcheng Yin X. Wang Jingwen Leng Zhouhan Lin ViT 29 11 0 24 May 2023
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model Xiao Wang Wei Zhou Qi Zhang Jie Zhou Songyang Gao Junzhe Wang Menghan Zhang Xiang Gao Yunwen Chen Tao Gui 38 7 0 22 May 2023
FIT: Far-reaching Interleaved Transformers Ting-Li Chen Lala Li 21 12 0 22 May 2023
AutoFocusFormer: Image Segmentation off the Grid Chen Ziwen K. Patnaik Shuangfei Zhai Alvin Wan Zhile Ren A. Schwing Alex Colburn Li Fuxin 17 9 0 24 Apr 2023
Dual-Attention Neural Transducers for Efficient Wake Word Spotting in Speech Recognition Saumya Yashmohini Sahai Jing Liu Thejaswi Muniyappa Kanthashree Mysore Sathyendra Anastasios Alexandridis ... Ross McGowan Ariya Rastrow Feng-Ju Chang Athanasios Mouchtaris Siegfried Kunzmann 31 5 0 03 Apr 2023
Transformers in Speech Processing: A Survey S. Latif Aun Zaidi Heriberto Cuayáhuitl Fahad Shamshad Moazzam Shoukat Junaid Qadir 42 47 0 21 Mar 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 26 18 0 09 Feb 2023
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers K. Choromanski Shanda Li Valerii Likhosherstov Kumar Avinava Dubey Shengjie Luo Di He Yiming Yang Tamás Sarlós Thomas Weingarten Adrian Weller 23 8 0 03 Feb 2023
Mnemosyne: Learning to Train Transformers with Transformers Deepali Jain K. Choromanski Kumar Avinava Dubey Sumeet Singh Vikas Sindhwani Tingnan Zhang Jie Tan OffRL 31 9 0 02 Feb 2023
Exploring Attention Map Reuse for Efficient Transformer Neural Networks Kyuhong Shim Jungwook Choi Wonyong Sung ViT 17 3 0 29 Jan 2023
Dynamic Grained Encoder for Vision Transformers Lin Song Songyang Zhang Songtao Liu Zeming Li Xuming He Hongbin Sun Jian-jun Sun Nanning Zheng ViT 26 34 0 10 Jan 2023
Unsupervised Manifold Linearizing and Clustering Tianjiao Ding Shengbang Tong Kwan Ho Ryan Chan Xili Dai Y. Ma B. Haeffele 16 11 0 04 Jan 2023
Efficient Long Sequence Modeling via State Space Augmented Transformer Simiao Zuo Xiaodong Liu Jian Jiao Denis Xavier Charles Eren Manavoglu Tuo Zhao Jianfeng Gao 120 36 0 15 Dec 2022
Meta-Learning Fast Weight Language Models Kevin Clark Kelvin Guu Ming-Wei Chang Panupong Pasupat Geoffrey E. Hinton Mohammad Norouzi KELM 27 13 0 05 Dec 2022
BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch? Joel Niklaus Daniele Giofré 27 11 0 30 Nov 2022
Efficient Transformers with Dynamic Token Pooling Piotr Nawrot J. Chorowski Adrian Lañcucki E. Ponti 14 42 0 17 Nov 2022
Language models are good pathologists: using attention-based sequence reduction and text-pretrained transformers for efficient WSI classification Juan Pisula Katarzyna Bozek VLM MedIm 28 3 0 14 Nov 2022
Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention Wenhao Li Xiaoyuan Yi Jinyi Hu Maosong Sun Xing Xie 21 0 0 14 Nov 2022
Efficiently Scaling Transformer Inference Reiner Pope Sholto Douglas Aakanksha Chowdhery Jacob Devlin James Bradbury Anselm Levskaya Jonathan Heek Kefan Xiao Shivani Agrawal J. Dean 32 292 0 09 Nov 2022
Graphically Structured Diffusion Models Christian Weilbach William Harvey Frank D. Wood DiffM 29 7 0 20 Oct 2022
Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation Botao Yu Peiling Lu Rui Wang Wei Hu Xu Tan Wei Ye Shikun Zhang Tao Qin Tie-Yan Liu MGen 25 54 0 19 Oct 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 39 9 0 14 Oct 2022