Understanding and Bridging the Modality Gap for Speech Translation

15 May 2023

Papers citing "Understanding and Bridging the Modality Gap for Speech Translation"

50 / 57 papers shown

Title
CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation Yan Zhou Qingkai Fang Yang Feng OT 111 30 0 24 May 2023
Back Translation for Speech-to-text Translation Without Transcripts Qingkai Fang Yang Feng 65 14 0 15 May 2023
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training Zi-Hua Zhang Long Zhou Junyi Ao Shujie Liu Lirong Dai Jinyu Li Furu Wei 121 58 0 07 Oct 2022
Cross-modal Contrastive Learning for Speech Translation Rong Ye Mingxuan Wang Lei Li SSL 76 91 0 05 May 2022
MAESTRO: Matched Speech Text Representations through Modality Matching Zhehuai Chen Yu Zhang Andrew Rosenberg Bhuvana Ramabhadran Pedro J. Moreno Ankur Bapna Heiga Zen 87 108 0 07 Apr 2022
Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation Kushal Arora Layla El Asri Hareesh Bahuleyan Jackie C.K. Cheung 81 82 0 03 Apr 2022
STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation Qingkai Fang Rong Ye Lei Li Yang Feng Mingxuan Wang 112 100 0 20 Mar 2022
Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation Tsz Kin Lam Shigehiko Schamoni Stefan Riezler 64 34 0 16 Mar 2022
mSLAM: Massively multilingual joint pre-training for speech and text Ankur Bapna Colin Cherry Yu Zhang Ye Jia Melvin Johnson Yong Cheng Simran Khanuja Jason Riesa Alexis Conneau VLM 67 114 0 03 Feb 2022
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale Arun Babu Changhan Wang Andros Tjandra Kushal Lakhotia Qiantong Xu ... Yatharth Saraf J. Pino Alexei Baevski Alexis Conneau Michael Auli SSL 112 709 0 17 Nov 2021
SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training Ankur Bapna Yu-An Chung Na Wu Anmol Gulati Ye Jia J. Clark Melvin Johnson Jason Riesa Alexis Conneau Yu Zhang VLM 131 96 0 20 Oct 2021
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task Yun Tang J. Pino Xian Li Changhan Wang Dmitriy Genzel 173 84 0 12 Jul 2021
R-Drop: Regularized Dropout for Neural Networks Xiaobo Liang Lijun Wu Juntao Li Yue Wang Qi Meng Tao Qin Wei Chen Hao Fei Tie-Yan Liu 84 437 0 28 Jun 2021
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units Wei-Ning Hsu Benjamin Bolte Yao-Hung Hubert Tsai Kushal Lakhotia Ruslan Salakhutdinov Abdel-rahman Mohamed SSL 188 3,004 0 14 Jun 2021
Lightweight Adapter Tuning for Multilingual Speech Translation Hang Le J. Pino Changhan Wang Jiatao Gu D. Schwab Laurent Besacier 120 90 0 02 Jun 2021
Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference? L. Bentivogli Mauro Cettolo Marco Gaido Alina Karakanta A. Martinelli Matteo Negri Marco Turchi 73 83 0 02 Jun 2021
Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation Yang Xu Yijin Liu Fandong Meng Jiajun Zhang Jinan Xu Jie Zhou 53 10 0 26 May 2021
Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders Chen Xu Bojie Hu Yanyang Li Yuhao Zhang Shen Huang Qi Ju Tong Xiao Jingbo Zhu 66 78 0 12 May 2021
Learning Shared Semantic Space for Speech-to-Text Translation Chi Han Mingxuan Wang Heng Ji Lei Li 94 78 0 07 May 2021
End-to-end Speech Translation via Cross-modal Progressive Training Rong Ye Mingxuan Wang Lei Li 71 73 0 21 Apr 2021
Large-Scale Self- and Semi-Supervised Learning for Speech Translation Changhan Wang Anne Wu J. Pino Alexei Baevski Michael Auli Alexis Conneau SSL 72 46 0 14 Apr 2021
Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation Hirofumi Inaguma Tatsuya Kawahara Shinji Watanabe 73 43 0 13 Apr 2021
Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation Renjie Zheng Junkun Chen Mingbo Ma Liang Huang 155 69 0 10 Feb 2021
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation Changhan Wang M. Rivière Ann Lee Anne Wu Chaitanya Talnikar Daniel Haziza Mary Williamson J. Pino Emmanuel Dupoux SSL 108 496 0 02 Jan 2021
MLS: A Large-Scale Multilingual Dataset for Speech Research Vineel Pratap Qiantong Xu Anuroop Sriram Gabriel Synnaeve R. Collobert AuLLM 115 512 0 07 Dec 2020
Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation Hang Le J. Pino Changhan Wang Jiatao Gu D. Schwab Laurent Besacier 100 83 0 02 Nov 2020
A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks Yun Tang J. Pino Changhan Wang Xutai Ma Dmitriy Genzel 71 75 0 21 Oct 2020
Token-level Adaptive Training for Neural Machine Translation Shuhao Gu Jinchao Zhang Fandong Meng Yang Feng Wanying Xie Jie Zhou Dong Yu 60 35 0 09 Oct 2020
Consecutive Decoding for Speech-to-text Translation Qianqian Dong Mingxuan Wang Hao Zhou Shuang Xu Bo Xu Lei Li SLR 95 41 0 21 Sep 2020
CoVoST 2 and Massively Multilingual Speech-to-Text Translation Changhan Wang Anne Wu J. Pino SLR 86 75 0 20 Jul 2020
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations Alexei Baevski Henry Zhou Abdel-rahman Mohamed Michael Auli SSL 303 5,853 0 20 Jun 2020
On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation Chaojun Wang Rico Sennrich 63 159 0 07 May 2020
Curriculum Pre-training for End-to-End Speech Translation Chengyi Wang Yu Wu Shujie Liu Ming Zhou Zhenglu Yang 77 109 0 21 Apr 2020
Speech Translation and the End-to-End Promise: Taking Stock of Where We Are Matthias Sperber Matthias Paulik 70 112 0 14 Apr 2020
Libri-Light: A Benchmark for ASR with Limited or No Supervision Jacob Kahn M. Rivière Weiyi Zheng Evgeny Kharitonov Qiantong Xu ... Tatiana Likhomanenko Gabriel Synnaeve Armand Joulin Abdel-rahman Mohamed Emmanuel Dupoux AuLLM 80 674 0 17 Dec 2019
Common Voice: A Massively-Multilingual Speech Corpus Rosana Ardila Megan Branson Kelly Davis Michael Henretty M. Kohler Josh Meyer Reuben Morais Lindsay Saunders Francis M. Tyers Gregor Weber VLM 100 1,622 0 13 Dec 2019
On Using SpecAugment for End-to-End Speech Translation Parnia Bahar Albert Zeyer Ralf Schluter Hermann Ney 85 54 0 20 Nov 2019
A Comparative Study on End-to-end Speech to Text Translation Parnia Bahar Tobias Bieschke Hermann Ney 87 80 0 20 Nov 2019
Analyzing ASR pretraining for low-resource speech-to-text translation Mihaela C. Stoian Sameer Bansal Sharon Goldwater 80 64 0 23 Oct 2019
Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation Chengyi Wang Yu-Huan Wu Shujie Liu Zhenglu Yang M. Zhou 82 84 0 17 Sep 2019
Bridging the Gap between Training and Inference for Neural Machine Translation Wen Zhang Yang Feng Fandong Meng Di You Qun Liu AIMat 80 242 0 06 Jun 2019
Fluent Translations from Disfluent Speech in End-to-End Speech Translation Elizabeth Salesky Matthias Sperber A. Waibel 66 34 0 03 Jun 2019
End-to-End Speech Translation with Knowledge Distillation Yuchen Liu Hao Xiong Zhongjun He Jiajun Zhang Hua Wu Haifeng Wang Chengqing Zong 92 155 0 17 Apr 2019
fairseq: A Fast, Extensible Toolkit for Sequence Modeling Myle Ott Sergey Edunov Alexei Baevski Angela Fan Sam Gross Nathan Ng David Grangier Michael Auli VLM FaML 132 3,159 0 01 Apr 2019
Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation Ye Jia Melvin Johnson Wolfgang Macherey Ron J. Weiss Yuan Cao Chung-Cheng Chiu Naveen Ari Stella Laurenzo Yonghui Wu 82 163 0 05 Nov 2018
Pre-training on high-resource speech recognition improves low-resource speech-to-text translation Sameer Bansal Herman Kamper Karen Livescu Adam Lopez Sharon Goldwater 78 197 0 05 Sep 2018
A Study of Reinforcement Learning for Neural Machine Translation Lijun Wu Fei Tian Tao Qin Jianhuang Lai Tie-Yan Liu OffRL 65 183 0 27 Aug 2018
A Call for Clarity in Reporting BLEU Scores Matt Post 187 2,998 0 23 Apr 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 819 132,725 0 12 Jun 2017
Sequence-to-Sequence Models Can Directly Translate Foreign Speech Ron J. Weiss J. Chorowski Navdeep Jaitly Yonghui Wu Zhiwen Chen 82 344 0 24 Mar 2017