Neural Architecture Search on Efficient Transformers and Beyond

28 July 2022

Zhen Qin

Papers citing "Neural Architecture Search on Efficient Transformers and Beyond"

20 / 20 papers shown

Title
AttentionSmithy: A Modular Framework for Rapid Transformer Development and Customization Caleb Cranney Jesse G. Meyer 85 0 0 13 Feb 2025
Scaling Laws for Linear Complexity Language Models Xuyang Shen Dong Li Ruitao Leng Zhen Qin Weigao Sun Yiran Zhong LRM 33 6 0 24 Jun 2024
Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention Zhen Qin Weigao Sun Dong Li Xuyang Shen Weixuan Sun Yiran Zhong 46 9 0 27 May 2024
Accelerating Toeplitz Neural Network with Constant-time Inference Complexity Zhen Qin Yiran Zhong 18 6 0 15 Nov 2023
Hierarchically Gated Recurrent Neural Network for Sequence Modeling Zhen Qin Songlin Yang Yiran Zhong 36 74 0 08 Nov 2023
Survey on Computer Vision Techniques for Internet-of-Things Devices Ishmeet Kaur Adwaita Janardhan Jadhav AI4CE 14 1 0 02 Aug 2023
TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer Zhen Qin Dong Li Weigao Sun Weixuan Sun Xuyang Shen ... Yunshen Wei Baohong Lv Xiao Luo Yu Qiao Yiran Zhong 43 15 0 27 Jul 2023
Exploring Transformer Extrapolation Zhen Qin Yiran Zhong Huiyuan Deng 31 9 0 19 Jul 2023
Linearized Relative Positional Encoding Zhen Qin Weixuan Sun Kaiyue Lu Huizhong Deng Dong Li Xiaodong Han Yuchao Dai Lingpeng Kong Yiran Zhong 20 12 0 18 Jul 2023
A Survey of Techniques for Optimizing Transformer Inference Krishna Teja Chitty-Venkata Sparsh Mittal M. Emani V. Vishwanath Arun Somani 40 62 0 16 Jul 2023
Improving Autoregressive NLP Tasks via Modular Linearized Attention Victor Agostinelli Lizhong Chen 22 1 0 17 Apr 2023
MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention Wenyuan Zeng Meng Li Wenjie Xiong Tong Tong Wen-jie Lu Jin Tan Runsheng Wang Ru Huang 24 20 0 25 Nov 2022
The Devil in Linear Transformer Zhen Qin Xiaodong Han Weixuan Sun Dongxu Li Lingpeng Kong Nick Barnes Yiran Zhong 36 70 0 19 Oct 2022
Linear Video Transformer with Feature Fixation Kaiyue Lu Zexia Liu Jianyuan Wang Weixuan Sun Zhen Qin ... Xuyang Shen Huizhong Deng Xiaodong Han Yuchao Dai Yiran Zhong 30 4 0 15 Oct 2022
DARTFormer: Finding The Best Type Of Attention Jason Brown Yiren Zhao Ilia Shumailov Robert D. Mullins 22 6 0 02 Oct 2022
Wide Attention Is The Way Forward For Transformers? Jason Brown Yiren Zhao Ilia Shumailov Robert D. Mullins 19 7 0 02 Oct 2022
Primer: Searching for Efficient Transformers for Language Modeling David R. So Wojciech Mañke Hanxiao Liu Zihang Dai Noam M. Shazeer Quoc V. Le VLM 85 152 0 17 Sep 2021
A Survey on Evolutionary Neural Architecture Search Yuqiao Liu Y. Sun Bing Xue Mengjie Zhang Gary G. Yen Kay Chen Tan 123 404 0 25 Aug 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 274 2,015 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 243 580 0 12 Mar 2020