Random Feature Attention

3 March 2021

Hao Peng

Lingpeng Kong

Papers citing "Random Feature Attention"

50 / 101 papers shown

Title
Conformal Transformations for Symmetric Power Transformers Saurabh Kumar Jacob Buckman Carles Gelada Sean Zhang 70 0 0 05 Mar 2025
Enhancing Video Understanding: Deep Neural Networks for Spatiotemporal Analysis Amir Hosein Fadaei M. Dehaqani 45 0 0 11 Feb 2025
Parallel Sequence Modeling via Generalized Spatial Propagation Network Hongjun Wang Wonmin Byeon Jiarui Xu Liang Feng Ka Chun Cheung Xiaolong Wang Kai Han Jan Kautz Sifei Liu 152 0 0 21 Jan 2025
Training Hybrid Neural Networks with Multimode Optical Nonlinearities Using Digital Twins Ilker Oguz Louis J. E. Suter J. Hsieh Mustafa Yildirim Niyazi Ulaş Dinç Christophe Moser D. Psaltis 58 2 0 14 Jan 2025
Key-value memory in the brain Samuel J. Gershman Ila Fiete Kazuki Irie 34 7 0 06 Jan 2025
Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass Tong Chen Hao Fang Patrick Xia Xiaodong Liu Benjamin Van Durme Luke Zettlemoyer Jianfeng Gao Hao Cheng KELM 53 2 0 08 Nov 2024
Estimating the Spectral Moments of the Kernel Integral Operator from Finite Sample Matrices Chanwoo Chun SueYeon Chung Daniel D. Lee 26 1 0 23 Oct 2024
An Evolved Universal Transformer Memory Edoardo Cetin Qi Sun Tianyu Zhao Yujin Tang 146 0 0 17 Oct 2024
Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity Mutian He Philip N. Garner 82 0 0 09 Oct 2024
Optimal Kernel Quantile Learning with Random Features Caixing Wang Xingdong Feng 42 0 0 24 Aug 2024
Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences Zicheng Liu Siyuan Li Li Wang Zedong Wang Yunfan Liu Stan Z. Li 35 7 0 12 Jun 2024
Linear Attention Sequence Parallelism Weigao Sun Zhen Qin Dong Li Xuyang Shen Yu Qiao Yiran Zhong 70 2 0 03 Apr 2024
DiJiang: Efficient Large Language Models through Compact Kernelization Hanting Chen Zhicheng Liu Xutao Wang Yuchuan Tian Yunhe Wang VLM 29 5 0 29 Mar 2024
Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling Mahdi Karami Ali Ghodsi VLM 48 6 0 28 Feb 2024
Zebra: Extending Context Window with Layerwise Grouped Local-Global Attention Kaiqiang Song Xiaoyang Wang Sangwoo Cho Xiaoman Pan Dong Yu 31 7 0 14 Dec 2023
Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Chenglu Zhu Jiatong Cai Sunyi Zheng Lin Yang VLM 35 4 0 21 Nov 2023
TorchDEQ: A Library for Deep Equilibrium Models Zhengyang Geng J. Zico Kolter VLM 56 12 0 28 Oct 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 28 15 0 28 Sep 2023
Lost in the Middle: How Language Models Use Long Contexts Nelson F. Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni Percy Liang RALM 40 1,404 0 06 Jul 2023
Spike-driven Transformer Man Yao Jiakui Hu Zhaokun Zhou Liuliang Yuan Yonghong Tian Boxing Xu Guoqi Li 34 114 0 04 Jul 2023
Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator Ziwei He Meng-Da Yang Minwei Feng Jingcheng Yin Xinbing Wang Jingwen Leng Zhouhan Lin ViT 35 11 0 24 May 2023
SKI to go Faster: Accelerating Toeplitz Neural Networks via Asymmetric Kernels Alexander Moreno Jonathan Mei Luke Walters 21 0 0 15 May 2023
Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens Zhanpeng Zeng Cole Hawkins Min-Fong Hong Aston Zhang Nikolaos Pappas Vikas Singh Shuai Zheng 21 6 0 07 May 2023
Improving Autoregressive NLP Tasks via Modular Linearized Attention Victor Agostinelli Lizhong Chen 22 1 0 17 Apr 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 30 41 0 07 Apr 2023
DIR-AS: Decoupling Individual Identification and Temporal Reasoning for Action Segmentation Peiyao Wang Haibin Ling 15 2 0 04 Apr 2023
An Evaluation of Memory Optimization Methods for Training Neural Networks Xiaoxuan Liu Siddharth Jha Alvin Cheung 26 0 0 26 Mar 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 34 18 0 09 Feb 2023
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers K. Choromanski Shanda Li Valerii Likhosherstov Kumar Avinava Dubey Shengjie Luo Di He Yiming Yang Tamás Sarlós Thomas Weingarten Adrian Weller 37 8 0 03 Feb 2023
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference Haoxin Li Phillip Keung Daniel Cheng Jungo Kasai Noah A. Smith 14 3 0 11 Jan 2023
Efficient Long Sequence Modeling via State Space Augmented Transformer Simiao Zuo Xiaodong Liu Jian Jiao Denis Xavier Charles Eren Manavoglu Tuo Zhao Jianfeng Gao 125 36 0 15 Dec 2022
Full Contextual Attention for Multi-resolution Transformers in Semantic Segmentation Loic Themyr Clément Rambour Nicolas Thome Toby Collins Alexandre Hostettler ViT 27 10 0 15 Dec 2022
LMEC: Learnable Multiplicative Absolute Position Embedding Based Conformer for Speech Recognition Yuguang Yang Y. Pan Jingjing Yin Heng Lu 24 3 0 05 Dec 2022
Token Turing Machines Michael S. Ryoo K. Gopalakrishnan Kumara Kahatapitiya Ted Xiao Kanishka Rao Austin Stone Yao Lu Julian Ibarz Anurag Arnab 27 21 0 16 Nov 2022
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers Michael Hassid Hao Peng Daniel Rotem Jungo Kasai Ivan Montero Noah A. Smith Roy Schwartz 32 24 0 07 Nov 2022
Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation Botao Yu Peiling Lu Rui Wang Wei Hu Xu Tan Wei Ye Shikun Zhang Tao Qin Tie-Yan Liu MGen 25 55 0 19 Oct 2022
The Devil in Linear Transformer Zhen Qin Xiaodong Han Weixuan Sun Dongxu Li Lingpeng Kong Nick Barnes Yiran Zhong 36 70 0 19 Oct 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 43 9 0 14 Oct 2022
LARF: Two-level Attention-based Random Forests with a Mixture of Contamination Models A. Konstantinov Lev V. Utkin 36 0 0 11 Oct 2022
Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules Kazuki Irie Jürgen Schmidhuber 37 5 0 07 Oct 2022
WavSpA: Wavelet Space Attention for Boosting Transformers' Long Sequence Learning Ability Yufan Zhuang Zihan Wang Fangbo Tao Jingbo Shang ViT AI4TS 35 3 0 05 Oct 2022
Transformer Meets Boundary Value Inverse Problems Ruchi Guo Shuhao Cao Long Chen MedIm 36 21 0 29 Sep 2022
Mega: Moving Average Equipped Gated Attention Xuezhe Ma Chunting Zhou Xiang Kong Junxian He Liangke Gui Graham Neubig Jonathan May Luke Zettlemoyer 14 183 0 21 Sep 2022
Adapting Pretrained Text-to-Text Models for Long Text Sequences Wenhan Xiong Anchit Gupta Shubham Toshniwal Yashar Mehdad Wen-tau Yih RALM VLM 54 30 0 21 Sep 2022
Efficient Methods for Natural Language Processing: A Survey Marcos Vinícius Treviso Ji-Ung Lee Tianchu Ji Betty van Aken Qingqing Cao ... Emma Strubell Niranjan Balasubramanian Leon Derczynski Iryna Gurevych Roy Schwartz 28 109 0 31 Aug 2022
A Circular Window-based Cascade Transformer for Online Action Detection Shuyuan Cao Weihua Luo Bairui Wang Wei Emma Zhang Lin Ma 42 6 0 30 Aug 2022
SpanDrop: Simple and Effective Counterfactual Learning for Long Sequences Peng Qi Guangtao Wang Jing Huang 24 0 0 03 Aug 2022
Efficient Long-Text Understanding with Short-Text Models Maor Ivgi Uri Shaham Jonathan Berant VLM 24 75 0 01 Aug 2022
Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization T. Nguyen Richard G. Baraniuk Robert M. Kirby Stanley J. Osher Bao Wang 26 9 0 01 Aug 2022
Neural Architecture Search on Efficient Transformers and Beyond Zexiang Liu Dong Li Kaiyue Lu Zhen Qin Weixuan Sun Jiacheng Xu Yiran Zhong 35 19 0 28 Jul 2022