v1v2 (latest)

Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget

9 September 2024

Andy T. Liu

Yi-Cheng Lin

Haibin Wu

Stefan Winkler

Hung-yi Lee

ArXiv (abs)PDF HTML

Papers citing "Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget"

34 / 34 papers shown

Title
Distilling a speech and music encoder with task arithmetic Fabian Ritter-Gutierrez Yi-Cheng Lin Jui-Chiang Wei Jeremy H.M Wong Eng Siong Chng Nancy F. Chen Hung-yi Lee 50 2 0 19 May 2025
Slamming: Training a Speech Language Model on One GPU in a Day Gallil Maimon Avishai Elmakies Yossi Adi 81 3 0 19 Feb 2025
Self-Supervised Speech Representations are More Phonetic than Semantic Kwanghee Choi Ankita Pasad Tomohiko Nakamura Satoru Fukayama Karen Livescu Shinji Watanabe 90 24 0 12 Jun 2024
Sustainable self-supervised learning for speech representations Luis Lugo Valentin Vielzeuf 69 2 0 11 Jun 2024
On the social bias of speech self-supervised models Yi-Cheng Lin Tzu-Quan Lin Hsi-Che Lin Andy T. Liu Hung-yi Lee 81 9 0 07 Jun 2024
A Large-Scale Evaluation of Speech Foundation Models Shu-Wen Yang Heng-Jui Chang Zili Huang Andy T. Liu Cheng-I Jeff Lai ... Kushal Lakhotia Shang-Wen Li Abdelrahman Mohamed Shinji Watanabe Hung-yi Lee 83 26 0 15 Apr 2024
Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces Oli Danyi Liu Hao Tang Sharon Goldwater SSL 66 14 0 21 May 2023
Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding Yifan Peng Kwangyoun Kim Felix Wu Prashant Sridhar Shinji Watanabe 69 39 0 27 Feb 2023
MelHuBERT: A simplified HuBERT on Mel spectrograms Tzu-Quan Lin Hung-yi Lee Hao Tang SSL 72 16 0 17 Nov 2022
Match to Win: Analysing Sequences Lengths for Efficient Self-supervised Learning in Speech and Audio Yan Gao Javier Fernandez-Marques Titouan Parcollet Pedro Porto Buarque de Gusmão Nicholas D. Lane 68 9 0 30 Sep 2022
Self-Supervised Speech Representation Learning: A Review Abdel-rahman Mohamed Hung-yi Lee Lasse Borgholt Jakob Drachmann Havtorn Joakim Edin ... Shang-Wen Li Karen Livescu Lars Maaløe Tara N. Sainath Shinji Watanabe SSL AI4TS 253 367 0 21 May 2022
On-demand compute reduction with stochastic wav2vec 2.0 Apoorv Vyas Wei-Ning Hsu Michael Auli Alexei Baevski 63 13 0 25 Apr 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 208 1,987 0 29 Mar 2022
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities Hsiang-Sheng Tsai Heng-Jui Chang Wen-Chin Huang Zili Huang Kushal Lakhotia ... Hsuan-Jui Chen Shang-Wen Li Shinji Watanabe Abdel-rahman Mohamed Hung-yi Lee 91 111 0 14 Mar 2022
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing Sanyuan Chen Chengyi Wang Zhengyang Chen Yu-Huan Wu Shujie Liu ... Yao Qian Jian Wu Micheal Zeng Xiangzhan Yu Furu Wei SSL 267 1,905 0 26 Oct 2021
Don't speak too fast: The impact of data bias on self-supervised speech models Yen Meng Yi-Hui Chou Andy T. Liu Hung-yi Lee 83 27 0 15 Oct 2021
DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT Heng-Jui Chang Shu-Wen Yang Hung-yi Lee SSL 124 175 0 05 Oct 2021
Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition Felix Wu Kwangyoun Kim Jing Pan Kyu Jeong Han Kilian Q. Weinberger Yoav Artzi 52 75 0 14 Sep 2021
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units Wei-Ning Hsu Benjamin Bolte Yao-Hung Hubert Tsai Kushal Lakhotia Ruslan Salakhutdinov Abdel-rahman Mohamed SSL 184 3,003 0 14 Jun 2021
PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition Cheng-I Jeff Lai Yang Zhang Alexander H. Liu Shiyu Chang Yi-Lun Liao Yung-Sung Chuang Kaizhi Qian Sameer Khurana David D. Cox James R. Glass VLM 144 77 0 10 Jun 2021
SUPERB: Speech processing Universal PERformance Benchmark Shu-Wen Yang Po-Han Chi Yung-Sung Chuang Cheng-I Jeff Lai Kushal Lakhotia ... Shuyan Dong Shang-Wen Li Shinji Watanabe Abdel-rahman Mohamed Hung-yi Lee SSL 111 943 0 03 May 2021
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies Alexander H. Liu Yu-An Chung James R. Glass SSL 80 88 0 01 Nov 2020
Similarity Analysis of Self-Supervised Speech Representations Yu-An Chung Yonatan Belinkov James R. Glass SSL 110 37 0 22 Oct 2020
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech Andy T. Liu Shang-Wen Li Hung-yi Lee SSL 132 359 0 12 Jul 2020
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations Alexei Baevski Henry Zhou Abdel-rahman Mohamed Michael Auli SSL 301 5,849 0 20 Jun 2020
Understanding Self-Attention of Self-Supervised Audio Transformers Shu-Wen Yang Andy T. Liu Hung-yi Lee 41 27 0 05 Jun 2020
Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation Po-Han Chi Pei-Hung Chung Tsung-Han Wu Chun-Cheng Hsieh Yen-Hao Chen Shang-Wen Li Hung-yi Lee SSL 76 148 0 18 May 2020
Vector-Quantized Autoregressive Predictive Coding Yu-An Chung Hao Tang James R. Glass SSL 43 115 0 17 May 2020
Unsupervised pretraining transfers well across languages M. Rivière Armand Joulin Pierre-Emmanuel Mazaré Emmanuel Dupoux SSL VLM 57 209 0 07 Feb 2020
Learning Robust and Multilingual Speech Representations Kazuya Kawakami Luyu Wang Chris Dyer Phil Blunsom Aaron van den Oord SSL 86 100 0 29 Jan 2020
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders Andy T. Liu Shu-Wen Yang Po-Han Chi Po-Chun Hsu Hung-yi Lee SSL 155 374 0 25 Oct 2019
An Unsupervised Autoregressive Model for Speech Representation Learning Yu-An Chung Wei-Ning Hsu Hao Tang James R. Glass SSL 91 409 0 05 Apr 2019
Representation Learning with Contrastive Predictive Coding Aaron van den Oord Yazhe Li Oriol Vinyals DRL SSL 354 10,369 0 10 Jul 2018
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 2.1K 150,433 0 22 Dec 2014