Combiner: Full Attention Transformer with Sparse Computation Cost

12 July 2021

Papers citing "Combiner: Full Attention Transformer with Sparse Computation Cost"

50 / 56 papers shown

Title
Dual Filter: A Mathematical Framework for Inference using Transformer-like Architectures Heng-Sheng Chang P. Mehta 39 0 0 01 May 2025
Random Long-Context Access for Mamba via Hardware-aligned Hierarchical Sparse Attention Xiang Hu Jiaqi Leng Jun Zhao Kewei Tu Wei Wu Mamba 50 0 0 23 Apr 2025
CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences Ziran Qin Yuchen Cao Mingbao Lin Wen Hu Shixuan Fan Ke Cheng Weiyao Lin Jianguo Li 71 3 0 16 Mar 2025
Reducing Reasoning Costs: The Path of Optimization for Chain of Thought via Sparse Attention Mechanism Libo Wang LRM AI4CE 48 0 0 14 Nov 2024
Prompt Compression for Large Language Models: A Survey Zongqian Li Yinhong Liu Yixuan Su Nigel Collier MQ 52 10 0 16 Oct 2024
InAttention: Linear Context Scaling for Transformers Joseph Eisner 21 0 0 09 Oct 2024
1.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Data Calvin Tan Jerome Wang ALM 38 2 0 07 Aug 2024
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers Chao Lou Zixia Jia Zilong Zheng Kewei Tu ODL 35 18 0 24 Jun 2024
Accelerating Transformers with Spectrum-Preserving Token Merging Hoai-Chau Tran D. M. Nguyen Duy M. Nguyen Trung Thanh Nguyen Ngan Le Pengtao Xie Daniel Sonntag James Y. Zou Binh T. Nguyen Mathias Niepert 42 8 0 25 May 2024
Socialized Learning: A Survey of the Paradigm Shift for Edge Intelligence in Networked Systems Xiaofei Wang Yunfeng Zhao Chao Qiu Qinghua Hu Victor C. M. Leung 35 6 0 20 Apr 2024
LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memory Zicheng Liu Li Wang Siyuan Li Zedong Wang Haitao Lin Stan Z. Li VLM 27 4 0 17 Apr 2024
State Space Model for New-Generation Network Alternative to Transformers: A Survey Tianlin Li Shiao Wang Yuhe Ding Yuehang Li Wentao Wu ... Bowei Jiang Chenglong Li Yaowei Wang Yonghong Tian Jin Tang Mamba 33 49 0 15 Apr 2024
Segmentation Guided Sparse Transformer for Under-Display Camera Image Restoration Jingyun Xue Tao Wang Jun Wang Kaihao Zhang ViT 51 2 0 09 Mar 2024
Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers Yiwei Lu Yaoliang Yu Xinlin Li Vahid Partovi Nia MQ 38 3 0 27 Feb 2024
Interactive Multi-Head Self-Attention with Linear Complexity Hankyul Kang Ming-Hsuan Yang Jongbin Ryu 21 1 0 27 Feb 2024
Model Compression and Efficient Inference for Large Language Models: A Survey Wenxiao Wang Wei Chen Yicong Luo Yongliu Long Zhengkai Lin Liye Zhang Binbin Lin Deng Cai Xiaofei He MQ 41 48 0 15 Feb 2024
SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design Seokju Yun Youngmin Ro ViT 44 29 0 29 Jan 2024
The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey Saurav Pawar S.M. Towhidul Islam Tonmoy S. M. M. Zaman Vinija Jain Aman Chadha Amitava Das 37 27 0 15 Jan 2024
SparQ Attention: Bandwidth-Efficient LLM Inference Luka Ribar Ivan Chelombiev Luke Hudlass-Galley Charlie Blake Carlo Luschi Douglas Orr 29 46 0 08 Dec 2023
Token Fusion: Bridging the Gap between Token Pruning and Token Merging Minchul Kim Shangqian Gao Yen-Chang Hsu Yilin Shen Hongxia Jin 23 29 0 02 Dec 2023
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey Yunpeng Huang Jingwei Xu Junyu Lai Zixu Jiang Taolue Chen ... Xiaoxing Ma Lijuan Yang Zhou Xin Shupeng Li Penghao Zhao LLMAG KELM 36 54 0 21 Nov 2023
Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability Jishnu Ray Chowdhury Cornelia Caragea 37 5 0 08 Nov 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 31 15 0 28 Sep 2023
Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers Matthew Dutson Yin Li M. Gupta ViT 43 8 0 25 Aug 2023
Attention Is Not All You Need Anymore Zhe Chen 29 3 0 15 Aug 2023
FLatten Transformer: Vision Transformer using Focused Linear Attention Dongchen Han Xuran Pan Yizeng Han Shiji Song Gao Huang 23 156 0 01 Aug 2023
Transformers in Reinforcement Learning: A Survey Pranav Agarwal A. Rahman P. St-Charles Simon J. D. Prince Samira Ebrahimi Kahou OffRL 24 18 0 12 Jul 2023
Extending Context Window of Large Language Models via Positional Interpolation Shouyuan Chen Sherman Wong Liangjian Chen Yuandong Tian 12 494 0 27 Jun 2023
Equivariant vs. Invariant Layers: A Comparison of Backbone and Pooling for Point Cloud Classification Abihith Kothapalli Ashkan Shahbazi Xinran Liu Robert Sheng Soheil Kolouri 3DPC 19 2 0 08 Jun 2023
The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles Md Shamim Hussain Mohammed J. Zaki D. Subramanian 37 3 0 02 Jun 2023
Hierarchical Attention Encoder Decoder Asier Mujika BDL 22 3 0 01 Jun 2023
Landmark Attention: Random-Access Infinite Context Length for Transformers Amirkeivan Mohtashami Martin Jaggi LLMAG 21 149 0 25 May 2023
FIT: Far-reaching Interleaved Transformers Ting-Li Chen Lala Li 26 12 0 22 May 2023
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers L. Yu Daniel Simig Colin Flaherty Armen Aghajanyan Luke Zettlemoyer M. Lewis 32 84 0 12 May 2023
EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention Xinyu Liu Houwen Peng Ningxin Zheng Yuqing Yang Han Hu Yixuan Yuan ViT 25 276 0 11 May 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 30 41 0 07 Apr 2023
AI-Generated Content (AIGC): A Survey Jiayang Wu Wensheng Gan Zefeng Chen Shicheng Wan Hong Lin 3DV 31 123 0 26 Mar 2023
A Unified View of Long-Sequence Models towards Modeling Million-Scale Dependencies Hongyu Hè Marko Kabić 25 2 0 13 Feb 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 34 18 0 09 Feb 2023
Efficient Transformers with Dynamic Token Pooling Piotr Nawrot J. Chorowski Adrian Lañcucki E. Ponti 20 42 0 17 Nov 2022
Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for Long Sequences Aosong Feng Irene Z Li Yuang Jiang Rex Ying 29 18 0 21 Oct 2022
Breaking BERT: Evaluating and Optimizing Sparsified Attention Siddhartha Brahma Polina Zablotskaia David M. Mimno 27 1 0 07 Oct 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra Christopher Ré VLM 75 2,024 0 27 May 2022
Your Transformer May Not be as Powerful as You Expect Shengjie Luo Shanda Li Shuxin Zheng Tie-Yan Liu Liwei Wang Di He 63 51 0 26 May 2022
Linear Complexity Randomized Self-attention Mechanism Lin Zheng Chong-Jun Wang Lingpeng Kong 22 31 0 10 Apr 2022
Memorizing Transformers Yuhuai Wu M. Rabe DeLesley S. Hutchins Christian Szegedy RALM 30 173 0 16 Mar 2022
Block-Recurrent Transformers DeLesley S. Hutchins Imanol Schlag Yuhuai Wu Ethan Dyer Behnam Neyshabur 23 94 0 11 Mar 2022
DCT-Former: Efficient Self-Attention with Discrete Cosine Transform Carmelo Scribano Giorgia Franchini M. Prato Marko Bertogna 18 21 0 02 Mar 2022
Transformer Quality in Linear Time Weizhe Hua Zihang Dai Hanxiao Liu Quoc V. Le 78 222 0 21 Feb 2022
General-purpose, long-context autoregressive modeling with Perceiver AR Curtis Hawthorne Andrew Jaegle Cătălina Cangea Sebastian Borgeaud C. Nash ... Hannah R. Sheahan Neil Zeghidour Jean-Baptiste Alayrac João Carreira Jesse Engel 43 65 0 15 Feb 2022