uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes

v1v2v3v4v5 (latest)

uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes

1 July 2024

Muhammad Abdul-Mageed

ArXiv (abs)PDF HTML

Papers citing "uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes"

13 / 13 papers shown

Title
To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation Abdul Waheed Karima Kadaoui Muhammad Abdul-Mageed VLM 63 3 0 06 Jun 2024
Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling Sanchit Gandhi Patrick von Platen Alexander M. Rush VLM 65 62 0 01 Nov 2023
SONAR: Sentence-Level Multimodal and Language-Agnostic Representations Paul-Ambroise Duquenne Holger Schwenk Benoît Sagot AI4TS VLM 107 70 0 22 Aug 2023
Robust Speech Recognition via Large-Scale Weak Supervision Alec Radford Jong Wook Kim Tao Xu Greg Brockman C. McLeavey Ilya Sutskever OffRL 215 3,750 0 06 Dec 2022
FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech Alexis Conneau Min Ma Simran Khanuja Yu Zhang Vera Axelrod Siddharth Dalmia Jason Riesa Clara E. Rivera Ankur Bapna VLM 147 329 0 25 May 2022
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale Arun Babu Changhan Wang Andros Tjandra Kushal Lakhotia Qiantong Xu ... Yatharth Saraf J. Pino Alexei Baevski Alexis Conneau Michael Auli SSL 110 708 0 17 Nov 2021
DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT Heng-Jui Chang Shu-Wen Yang Hung-yi Lee SSL 121 175 0 05 Oct 2021
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units Wei-Ning Hsu Benjamin Bolte Yao-Hung Hubert Tsai Kushal Lakhotia Ruslan Salakhutdinov Abdel-rahman Mohamed SSL 184 3,003 0 14 Jun 2021
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic Muhammad Abdul-Mageed AbdelRahim Elmadany El Moatez Billah Nagoudi VLM 112 464 0 27 Dec 2020
Common Voice: A Massively-Multilingual Speech Corpus Rosana Ardila Megan Branson Kelly Davis Michael Henretty M. Kohler Josh Meyer Reuben Morais Lindsay Saunders Francis M. Tyers Gregor Weber VLM 93 1,620 0 13 Dec 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 255 7,554 0 02 Oct 2019
Data-Free Knowledge Distillation for Deep Neural Networks Raphael Gontijo-Lopes Stefano Fenu Thad Starner 62 273 0 19 Oct 2017
Speech Recognition Challenge in the Wild: Arabic MGB-3 Ahmed M. Ali S. Vogel Steve Renals 56 104 0 21 Sep 2017