Linear Attention via Orthogonal Memory

18 December 2023

Lingpeng Kong

Papers citing "Linear Attention via Orthogonal Memory"

25 / 25 papers shown

Title
In-Context Learning with Many Demonstration Examples Mukai Li Shansan Gong Jiangtao Feng Yiheng Xu Jinchao Zhang Zhiyong Wu Lingpeng Kong 80 38 0 09 Feb 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 81 20 0 09 Feb 2023
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Dianbo Sui 3DV 108 9 0 14 Oct 2022
On the Parameterization and Initialization of Diagonal State Space Models Albert Gu Ankit Gupta Karan Goel Christopher Ré 81 315 0 23 Jun 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 471 6,231 0 05 Apr 2022
cosFormer: Rethinking Softmax in Attention Zhen Qin Weixuan Sun Huicai Deng Dongxu Li Yunshen Wei Baohong Lv Junjie Yan Lingpeng Kong Yiran Zhong 65 219 0 17 Feb 2022
XCiT: Cross-Covariance Image Transformers Alaaeldin El-Nouby Hugo Touvron Mathilde Caron Piotr Bojanowski Matthijs Douze ... Ivan Laptev Natalia Neverova Gabriel Synnaeve Jakob Verbeek Hervé Jégou ViT 136 511 0 17 Jun 2021
Luna: Linear Unified Nested Attention Xuezhe Ma Xiang Kong Sinong Wang Chunting Zhou Jonathan May Hao Ma Luke Zettlemoyer 80 113 0 03 Jun 2021
RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su Yu Lu Shengfeng Pan Ahmed Murtadha Bo Wen Yunfeng Liu 268 2,443 0 20 Apr 2021
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin B. Guo ViT 441 21,418 0 25 Mar 2021
GLM: General Language Model Pretraining with Autoregressive Blank Infilling Zhengxiao Du Yujie Qian Xiao Liu Ming Ding J. Qiu Zhilin Yang Jie Tang BDL AI4CE 125 1,545 0 18 Mar 2021
Long Range Arena: A Benchmark for Efficient Transformers Yi Tay Mostafa Dehghani Samira Abnar Songlin Yang Dara Bahri Philip Pham J. Rao Liu Yang Sebastian Ruder Donald Metzler 136 718 0 08 Nov 2020
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 179 1,580 0 30 Sep 2020
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding Shuohang Wang Luowei Zhou Zhe Gan Yen-Chun Chen Yuwei Fang S. Sun Yu Cheng Jingjing Liu 70 28 0 13 Sep 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 540 2,081 0 28 Jul 2020
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos Apoorv Vyas Nikolaos Pappas Franccois Fleuret 201 1,765 0 29 Jun 2020
Linformer: Self-Attention with Linear Complexity Sinong Wang Belinda Z. Li Madian Khabsa Han Fang Hao Ma 210 1,702 0 08 Jun 2020
Longformer: The Long-Document Transformer Iz Beltagy Matthew E. Peters Arman Cohan RALM VLM 162 4,062 0 10 Apr 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 307 596 0 12 Mar 2020
PF-Net: Point Fractal Network for 3D Point Cloud Completion Zitian Huang Yikuan Yu Jiawen Xu Feng Ni Xinyi Le 3DPC 97 359 0 01 Mar 2020
A review on deep learning techniques for 3D sensed data classification David J. Griffiths Jan Boehm 3DPC 3DV 76 165 0 09 Jul 2019
What Do Single-view 3D Reconstruction Networks Learn? Maxim Tatarchenko Stephan R. Richter René Ranftl Zhuwen Li V. Koltun Thomas Brox 3DV 50 427 0 09 May 2019
Image Transformer Niki Parmar Ashish Vaswani Jakob Uszkoreit Lukasz Kaiser Noam M. Shazeer Alexander Ku Dustin Tran ViT 133 1,679 0 15 Feb 2018
Effective Approaches to Attention-based Neural Machine Translation Thang Luong Hieu H. Pham Christopher D. Manning 377 7,962 0 17 Aug 2015
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks Andrew M. Saxe James L. McClelland Surya Ganguli ODL 165 1,844 0 20 Dec 2013