Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity

9 October 2024

Papers citing "Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity"

50 / 56 papers shown

Title
Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models Aviv Bick Kevin Y. Li Eric P. Xing J. Zico Kolter Albert Gu Mamba 92 28 0 19 Aug 2024
Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations Sarthak Yadav Zheng-Hua Tan Mamba 60 16 0 04 Jun 2024
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality Tri Dao Albert Gu Mamba 68 489 0 31 May 2024
Mamba in Speech: Towards an Alternative to Self-Attention Xiangyu Zhang Qiquan Zhang Hexin Liu Tianyi Xiao Xinyuan Qian Beena Ahmed E. Ambikairajah Haizhou Li Julien Epps Mamba 88 42 0 21 May 2024
SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model Siavash Shams Sukru Samet Dindar Xilin Jiang N. Mesgarani Mamba 84 21 0 20 May 2024
Linearizing Large Language Models Jean Mercat Igor Vasiljevic Sedrick Scott Keh Kushal Arora Achal Dave Adrien Gaidon Thomas Kollar 81 21 0 10 May 2024
Cross-Architecture Transfer Learning for Linear-Cost Inference Transformers Sehyun Choi 54 3 0 03 Apr 2024
SPMamba: State-space model is all you need in speech separation Kai Li Guo Chen Mamba 71 0 0 02 Apr 2024
Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation Xilin Jiang Cong Han N. Mesgarani Mamba 81 45 0 27 Mar 2024
The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry Michael Zhang Kush S. Bhatia Hermann Kumbong Christopher Ré 59 52 0 06 Feb 2024
Gated Linear Attention Transformers with Hardware-Efficient Training Aaron Courville Bailin Wang Songlin Yang Yikang Shen Yoon Kim 69 167 0 11 Dec 2023
Mamba: Linear-Time Sequence Modeling with Selective State Spaces Albert Gu Tri Dao Mamba 118 2,636 0 01 Dec 2023
Efficient Transformer Knowledge Distillation: A Performance Review Nathan Brown Ashton Williamson Tahj Anderson Logan Lawrence VLM 33 5 0 22 Nov 2023
Exploring RWKV for Memory Efficient and Low Latency Streaming ASR Keyu An Shiliang Zhang 79 4 0 26 Sep 2023
Investigating End-to-End ASR Architectures for Long Form Audio Transcription Nithin Rao Koluguri Samuel Kriman Georgy Zelenfroind Somshubra Majumdar Dima Rekesh Vahid Noroozi Jagadeesh Balam Boris Ginsburg AuLLM 62 9 0 18 Sep 2023
Continuation Path Learning for Homotopy Optimization Xi Lin Zhiyuan Yang Xiao-Yan Zhang Qingfu Zhang 78 9 0 24 Jul 2023
Retentive Network: A Successor to Transformer for Large Language Models Yutao Sun Li Dong Shaohan Huang Shuming Ma Yuqing Xia Jilong Xue Jianyong Wang Furu Wei LRM 95 328 0 17 Jul 2023
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation Yixiao Li Yifan Yu Qingru Zhang Chen Liang Pengcheng He Weizhu Chen Tuo Zhao 91 73 0 20 Jun 2023
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai 63 657 0 22 May 2023
RWKV: Reinventing RNNs for the Transformer Era Bo Peng Eric Alcaide Quentin G. Anthony Alon Albalak Samuel Arcadinho ... Qihang Zhao P. Zhou Qinghua Zhou Jian Zhu Rui-Jie Zhu 183 590 0 22 May 2023
The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation Mutian He Philip N. Garner 68 4 0 16 May 2023
Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition Dima Rekesh Nithin Rao Koluguri Samuel Kriman Somshubra Majumdar Vahid Noroozi ... Oleksii Hrinchuk Krishna Puvvada Ankur Kumar Jagadeesh Balam Boris Ginsburg 72 91 0 08 May 2023
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling Stella Biderman Hailey Schoelkopf Quentin G. Anthony Herbie Bradley Kyle O'Brien ... USVSN Sai Prashanth Edward Raff Aviya Skowron Lintang Sutawika Oskar van der Wal 87 1,268 0 03 Apr 2023
Resurrecting Recurrent Neural Networks for Long Sequences Antonio Orvieto Samuel L. Smith Albert Gu Anushan Fernando Çağlar Gülçehre Razvan Pascanu Soham De 259 287 0 11 Mar 2023
Fine-Tuning Pre-trained Transformers into Decaying Fast Weights H. H. Mao 89 22 0 09 Oct 2022
Efficiently Modeling Long Sequences with Structured State Spaces Albert Gu Karan Goel Christopher Ré 182 1,761 0 31 Oct 2021
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers Albert Gu Isys Johnson Karan Goel Khaled Kamal Saab Tri Dao Atri Rudra Christopher Ré 100 587 0 26 Oct 2021
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing Sanyuan Chen Chengyi Wang Zhengyang Chen Yu-Huan Wu Shujie Liu ... Yao Qian Jian Wu Micheal Zeng Xiangzhan Yu Furu Wei SSL 206 1,846 0 26 Oct 2021
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units Wei-Ning Hsu Benjamin Bolte Yao-Hung Hubert Tsai Kushal Lakhotia Ruslan Salakhutdinov Abdel-rahman Mohamed SSL 145 2,939 0 14 Jun 2021
An Attention Free Transformer Shuangfei Zhai Walter A. Talbott Nitish Srivastava Chen Huang Hanlin Goh Ruixiang Zhang J. Susskind ViT 59 130 0 28 May 2021
SUPERB: Speech processing Universal PERformance Benchmark Shu-Wen Yang Po-Han Chi Yung-Sung Chuang Cheng-I Jeff Lai Kushal Lakhotia ... Shuyan Dong Shang-Wen Li Shinji Watanabe Abdel-rahman Mohamed Hung-yi Lee SSL 90 929 0 03 May 2021
Annealing Knowledge Distillation A. Jafari Mehdi Rezagholizadeh Pranav Sharma A. Ghodsi 45 79 0 14 Apr 2021
Finetuning Pretrained Transformers into RNNs Jungo Kasai Hao Peng Yizhe Zhang Dani Yogatama Gabriel Ilharco Nikolaos Pappas Yi Mao Weizhu Chen Noah A. Smith 81 65 0 24 Mar 2021
Random Feature Attention Hao Peng Nikolaos Pappas Dani Yogatama Roy Schwartz Noah A. Smith Lingpeng Kong 80 356 0 03 Mar 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 425 2,081 0 31 Dec 2020
SLURP: A Spoken Language Understanding Resource Package E. Bastianelli Andrea Vanzo P. Swietojanski Verena Rieser VLM 85 228 0 26 Nov 2020
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 144 1,115 0 14 Sep 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 499 2,074 0 28 Jul 2020
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos Apoorv Vyas Nikolaos Pappas Franccois Fleuret 166 1,755 0 29 Jun 2020
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations Alexei Baevski Henry Zhou Abdel-rahman Mohamed Michael Auli SSL 219 5,767 0 20 Jun 2020
Linformer: Self-Attention with Linear Complexity Sinong Wang Belinda Z. Li Madian Khabsa Han Fang Hao Ma 183 1,694 0 08 Jun 2020
Longformer: The Long-Document Transformer Iz Beltagy Matthew E. Peters Arman Cohan RALM VLM 124 4,048 0 10 Apr 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 294 596 0 12 Mar 2020
Reformer: The Efficient Transformer Nikita Kitaev Lukasz Kaiser Anselm Levskaya VLM 176 2,307 0 13 Jan 2020
Knowledge Distillation from Internal Representations Gustavo Aguilar Yuan Ling Yu Zhang Benjamin Yao Xing Fan Edward Guo 64 181 0 08 Oct 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 183 7,465 0 02 Oct 2019
Patient Knowledge Distillation for BERT Model Compression S. Sun Yu Cheng Zhe Gan Jingjing Liu 114 836 0 25 Aug 2019
Generating Long Sequences with Sparse Transformers R. Child Scott Gray Alec Radford Ilya Sutskever 90 1,894 0 23 Apr 2019
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks Raphael Tang Yao Lu Linqing Liu Lili Mou Olga Vechtomova Jimmy J. Lin 61 419 0 28 Mar 2019
DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks Xingjian Li Haoyi Xiong Hanchao Wang Yuxuan Rao Liping Liu Zeyu Chen Jun Huan 51 172 0 26 Jan 2019