Title
On The Landscape of Spoken Language Models: A Comprehensive Survey Siddhant Arora Kai-Wei Chang Chung-Ming Chien Yifan Peng Haibin Wu Yossi Adi Emmanuel Dupoux Hung-yi Lee Karen Livescu Shinji Watanabe 52 2 0 11 Apr 2025
TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling Liang-Hsuan Tseng Yi-Chang Chen Kuan-Yi Lee Da-shan Shiu Hung-yi Lee AuLLM 56 0 0 09 Apr 2025
Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics Siddhant Arora Zhiyun Lu Chung-Cheng Chiu Ruoming Pang Shinji Watanabe 43 2 0 03 Mar 2025
AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues Se Jin Park Yeonju Kim Hyeongseop Rha Bella Godiva Y. Ro 36 1 0 23 Dec 2024
Building a Taiwanese Mandarin Spoken Language Model: A First Attempt Chih-Kai Yang Yu-Kuan Fu Chen An Li Yi-Cheng Lin Yu-Xiang Lin ... Ulin Sanga Xuanjun Chen Po-Chun Hsu Shu-Wen Yang Hung-yi Lee AuLLM 46 0 0 11 Nov 2024
Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant Alan Dao Dinh Bach Vu Huy Hoang Ha AuLLM VLM 73 3 0 20 Oct 2024
DM-Codec: Distilling Multimodal Representations for Speech Tokenization Md Mubtasim Ahasan Md Fahim Tasnim Mohiuddin A K M Mahbubur Rahman Aman Chadha Tariq Iqbal M. A. Amin Md. Mofijul Islam Amin Ahsan Ali 30 0 0 19 Oct 2024
Recent Advances in Speech Language Models: A Survey Wenqian Cui Dianzhi Yu Xiaoqi Jiao Ziqiao Meng Guangyan Zhang Qichao Wang Yiwen Guo Irwin King AuLLM 61 14 0 01 Oct 2024
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions Kai Chen Yunhao Gou Runhui Huang Zhili Liu Daxin Tan ... Qun Liu Jun Yao Lu Hou Hang Xu Hang Xu AuLLM MLLM VLM 82 21 0 26 Sep 2024
Speech Recognition Rescoring with Large Speech-Text Foundation Models Prashanth Gurunath Shivakumar J. Kolehmainen Aditya Gourav Yi Gu Ankur Gandhe Ariya Rastrow I. Bulyko AuLLM 31 0 0 25 Sep 2024
Salmon: A Suite for Acoustic Language Model Evaluation Gallil Maimon Amit Roth Yossi Adi ELM AuLLM 51 5 0 11 Sep 2024
LLaMA-Omni: Seamless Speech Interaction with Large Language Models Qingkai Fang Shoutao Guo Yan Zhou Zhengrui Ma Shaolei Zhang Yang Feng AuLLM 27 30 0 10 Sep 2024
LAST: Language Model Aware Speech Tokenization A. Turetzky Yossi Adi 37 2 0 05 Sep 2024
Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation Yinghao Aaron Li Xilin Jiang Jordan Darefsky Ge Zhu N. Mesgarani 41 2 0 13 Aug 2024
Language Model Can Listen While Speaking Ziyang Ma Yakun Song Chenpeng Du Jian Cong Zhuo Chen Yuping Wang Yue Wang Xie Chen AuLLM 37 23 0 05 Aug 2024
Investigating the Effects of Large-Scale Pseudo-Stereo Data and Different Speech Foundation Model on Dialogue Generative Spoken Language Model Yu-Kuan Fu Cheng-Kuang Lee Hsiu-Hsuan Wang Hung-yi Lee 24 0 0 02 Jul 2024
NAST: Noise Aware Speech Tokenization for Speech Language Models Shoval Messica Yossi Adi 30 6 0 16 Jun 2024
The Interspeech 2024 Challenge on Speech Processing Using Discrete Units Xuankai Chang Jiatong Shi Jinchuan Tian Yuning Wu Yuxun Tang Yihan Wu Shinji Watanabe Yossi Adi Xie Chen Qin Jin 47 15 0 11 Jun 2024
Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning Chung-Ming Chien Andros Tjandra Apoorv Vyas Matt Le Bowen Shi Wei-Ning Hsu 32 0 0 10 Jun 2024
A Survey of Deep Learning Audio Generation Methods Matej Bozic Marko Horvat VLM MedIm 54 0 0 31 May 2024
Zipper: A Multi-Tower Decoder Architecture for Fusing Modalities Vicky Zayats Peter Chen Melissa Ferrari Dirk Padfield AI4CE 38 0 0 29 May 2024
TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation Chenyang Le Yao Qian Dongmei Wang Long Zhou Shujie Liu ... Midia Yousefi Yanmin Qian Jinyu Li Sheng Zhao Michael Zeng 41 3 0 28 May 2024
CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations Leying Zhang Yao Qian Long Zhou Shujie Liu Dongmei Wang ... Yanmin Qian Jinyu Li Lei He Sheng Zhao Michael Zeng 34 1 0 10 Apr 2024
CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech Jaehyeon Kim Keon Lee Seungjun Chung Jaewoong Cho 74 39 0 03 Apr 2024
A Collection of Pragmatic-Similarity Judgments over Spoken Dialog Utterances Nigel G. Ward Divette Marco 26 5 0 21 Mar 2024
Towards audio language modeling -- an overview Haibin Wu Xuanjun Chen Yi-Cheng Lin Kai-Wei Chang Ho-Lam Chung Alexander H. Liu Hung-yi Lee AuLLM 35 28 0 20 Feb 2024
Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations Jaeyeon Kim Injune Hwang Kyogu Lee 19 0 0 02 Feb 2024
Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation Minsu Kim Jeong Hun Yeo Se Jin Park J. Choi Y. Ro 27 5 0 18 Jan 2024
Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue Guan-Ting Lin Prashanth Gurunath Shivakumar Ankur Gandhe Chao-Han Huck Yang Yile Gu Shalini Ghosh A. Stolcke Hung-yi Lee I. Bulyko 27 12 0 23 Dec 2023
TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation Xize Cheng Rongjie Huang Linjun Li Tao Jin Zehan Wang Aoxiong Yin Minglei Li Xinyu Duan Changpeng Yang Zhou Zhao 30 2 0 23 Dec 2023
AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation J. Choi Se Jin Park Minsu Kim Y. Ro 33 12 0 05 Dec 2023
Generative Pre-training for Speech with Flow Matching Alexander H. Liu Matt Le Apoorv Vyas Bowen Shi Andros Tjandra Wei-Ning Hsu 24 31 0 25 Oct 2023
Generative Spoken Language Model based on continuous word-sized audio tokens Robin Algayres Yossi Adi Tu Nguyen Jade Copet Gabriel Synnaeve Benoît Sagot Emmanuel Dupoux AuLLM 40 12 0 08 Oct 2023
Towards human-like spoken dialogue generation between AI agents from written dialogue Kentaro Mitsui Yukiya Hono Kei Sawada 31 13 0 02 Oct 2023
Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens Minsu Kim J. Choi Soumi Maiti Jeong Hun Yeo Shinji Watanabe Y. Ro VLM 26 6 0 15 Sep 2023
Sparks of Large Audio Models: A Survey and Outlook S. Latif Moazzam Shoukat Fahad Shamshad Muhammad Usama Yi Ren ... Wenwu Wang Xulong Zhang Roberto Togneri Erik Cambria Björn W. Schuller LM&MA AuLLM 33 38 0 24 Aug 2023
Many-to-Many Spoken Language Translation via Unified Speech and Text Representation Learning with Unit-to-Unit Translation Minsu Kim J. Choi Dahun Kim Y. Ro 40 10 0 03 Aug 2023
What Do Self-Supervised Speech Models Know About Words? Ankita Pasad C. Chien Shane Settle Karen Livescu SSL 37 26 0 30 Jun 2023
Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale Matt Le Apoorv Vyas Bowen Shi Brian Karrer Leda Sari ... Mary Williamson Vimal Manohar Yossi Adi Jay Mahadeokar Wei-Ning Hsu AuLLM 28 265 0 23 Jun 2023
Intelligible Lip-to-Speech Synthesis with Speech Units J. Choi Minsu Kim Y. Ro 26 24 0 31 May 2023
Make-A-Voice: Unified Voice Synthesis With Discrete Representation Rongjie Huang Chunlei Zhang Yongqiang Wang Dongchao Yang Lu Liu Zhenhui Ye Ziyue Jiang Chao Weng Zhou Zhao Dong Yu DiffM 31 26 0 30 May 2023
Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM Eliya Nachmani Alon Levkovitch Roy Hirsch Julián Salazar Chulayutsh Asawaroengchai Soroosh Mariooryad Ehud Rivlin RJ Skerry-Ryan Michelle Tadmor Ramanovich AuLLM 28 30 0 24 May 2023
Textually Pretrained Speech Language Models Michael Hassid Tal Remez Tu Nguyen Itai Gat Alexis Conneau ... Alexandre Défossez Gabriel Synnaeve Emmanuel Dupoux Roy Schwartz Yossi Adi VLM SyDa 31 53 0 22 May 2023
Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model Puyuan Peng Shang-Wen Li Okko Rasanen Abdel-rahman Mohamed David Harwath SSL VLM 26 7 0 19 May 2023
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities Dong Zhang Shimin Li Xin Zhang Jun Zhan Pengyu Wang Yaqian Zhou Xipeng Qiu AuLLM MLLM 62 293 0 18 May 2023
Back Translation for Speech-to-text Translation Without Transcripts Qingkai Fang Yang Feng 35 13 0 15 May 2023
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head Rongjie Huang Mingze Li Dongchao Yang Jiatong Shi Xuankai Chang ... Jia-Bin Huang Jinglin Liu Yixiang Ren Zhou Zhao Shinji Watanabe LM&MA AuLLM 42 198 0 25 Apr 2023
disco: a toolkit for Distributional Control of Generative Models Germán Kruszewski Jos Rozen Marc Dymetman 24 4 0 08 Mar 2023
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision Eugene Kharitonov Damien Vincent Zalan Borsos Raphaël Marinier Sertan Girgin Olivier Pietquin Matthew Sharifi Marco Tagliasacchi Neil Zeghidour 15 189 0 07 Feb 2023
Open Problems in Applied Deep Learning M. Raissi AI4CE 42 2 0 26 Jan 2023