Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks

14 September 2023

Papers citing "Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks"

48 / 48 papers shown

Title
Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play Yemin Shi Yu Shu Siwei Dong Guangyi Liu Jaward Sesay Jingwen Li Zhiting Hu AuLLM VLM 50 0 0 05 May 2025
On The Landscape of Spoken Language Models: A Comprehensive Survey Siddhant Arora Kai-Wei Chang Chung-Ming Chien Yifan Peng Haibin Wu Yossi Adi Emmanuel Dupoux Hung-yi Lee Karen Livescu Shinji Watanabe 52 2 0 11 Apr 2025
TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling Liang-Hsuan Tseng Yi-Chang Chen Kuan-Yi Lee Da-shan Shiu Hung-yi Lee AuLLM 56 0 0 09 Apr 2025
From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM Kshitij Ambilduke Ben Peters Sonal Sannigrahi Anil Keshwani Tsz Kin Lam Bruno Martins Marcely Zanon Boito André F. T. Martins 52 0 0 13 Mar 2025
Discrete Speech Unit Extraction via Independent Component Analysis Tomohiko Nakamura Kwanghee Choi Keigo Hojo Yoshiaki Bando Satoru Fukayama Shinji Watanabe 43 0 0 11 Jan 2025
SSR-Speech: Towards Stable, Safe and Robust Zero-shot Text-based Speech Editing and Synthesis Helin Wang Meng Yu Jiarui Hai Chen Chen Yuchen Hu Rilin Chen Najim Dehak Dong Yu 87 3 0 03 Jan 2025
Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognition Yoshiki Masuyama Koichi Miyazaki Masato Murata Mamba 43 0 0 11 Nov 2024
Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback Guan-Ting Lin Prashanth Gurunath Shivakumar Aditya Gourav Yile Gu Ankur Gandhe Hung-yi Lee I. Bulyko 29 8 0 04 Nov 2024
DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models Heng-Jui Chang Hongyu Gong Changhan Wang James R. Glass Yu-An Chung 28 0 0 31 Oct 2024
Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation Maohao Shen Shun Zhang Jilong Wu Zhiping Xiu Ehab AlBadawy Yiting Lu M. Seltzer Qing He 36 2 0 27 Oct 2024
VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning Yifan Peng Krishna C. Puvvada Zhehuai Chen Piotr .Zelasko He Huang Kunal Dhawan Ke Hu Shinji Watanabe Jagadeesh Balam Boris Ginsburg 58 2 0 23 Oct 2024
Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant Alan Dao Dinh Bach Vu Huy Hoang Ha AuLLM VLM 73 3 0 20 Oct 2024
Roadmap towards Superhuman Speech Understanding using Large Language Models Fan Bu Yuhao Zhang Xuben Wang Benyou Wang Qiang Liu Hao Li LM&MA ELM AuLLM 141 1 0 17 Oct 2024
FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Model Yichen Lu Jiaqi Song Chao-Han Huck Yang Shinji Watanabe 23 0 0 03 Oct 2024
Efficient Streaming LLM for Speech Recognition J. Jia Gil Keren Wei Zhou Egor Lakomkin Xiaohui Zhang Chunyang Wu Frank Seide Jay Mahadeokar Ozlem Kalinli AuLLM 29 0 0 02 Oct 2024
Recent Advances in Speech Language Models: A Survey Wenqian Cui Dianzhi Yu Xiaoqi Jiao Ziqiao Meng Guangyan Zhang Qichao Wang Yiwen Guo Irwin King AuLLM 59 14 0 01 Oct 2024
Speech Recognition Rescoring with Large Speech-Text Foundation Models Prashanth Gurunath Shivakumar J. Kolehmainen Aditya Gourav Yi Gu Ankur Gandhe Ariya Rastrow I. Bulyko AuLLM 28 0 0 25 Sep 2024
Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents Bandhav Veluri Benjamin Peloquin Bokai Yu Hongyu Gong Shyamnath Gollakota AuLLM OffRL 55 13 0 23 Sep 2024
Preference Alignment Improves Language Model-Based TTS Jinchuan Tian Chunlei Zhang Jiatong Shi Hao Zhang Jianwei Yu Shinji Watanabe Dong Yu 32 7 0 19 Sep 2024
Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach Maxime Poli Emmanuel Chemla Emmanuel Dupoux 34 2 0 16 Sep 2024
ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration Masao Someki Kwanghee Choi Siddhant Arora William Chen Samuele Cornell Jionghao Han Yifan Peng Jiatong Shi Vaibhav Srivastav Shinji Watanabe VLM 32 0 0 14 Sep 2024
Salmon: A Suite for Acoustic Language Model Evaluation Gallil Maimon Amit Roth Yossi Adi ELM AuLLM 51 5 0 11 Sep 2024
SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data Yichen Lu Álvaro Huertas-García Xuankai Chang Hengwei Bian Soumi Maiti Shinji Watanabe 46 2 0 01 Aug 2024
dMel: Speech Tokenization made Simple Richard He Bai Tatiana Likhomanenko Ruixiang Zhang Zijin Gu Zakaria Aldeneh Navdeep Jaitly 37 4 0 22 Jul 2024
On the Effectiveness of Acoustic BPE in Decoder-Only TTS Bohan Li Feiyu Shen Yiwei Guo Shuai Wang Xie Chen Kai Yu 39 2 0 04 Jul 2024
Towards Robust Speech Representation Learning for Thousands of Languages William Chen Wangyou Zhang Yifan Peng Xinjian Li Jinchuan Tian Jiatong Shi Xuankai Chang Soumi Maiti Karen Livescu Shinji Watanabe ELM 42 6 0 30 Jun 2024
Decoder-only Architecture for Streaming End-to-end Speech Recognition E. Tsunoo Hayato Futami Yosuke Kashiwagi Siddhant Arora Shinji Watanabe RALM AuLLM 36 6 0 23 Jun 2024
MMM: Multi-Layer Multi-Residual Multi-Stream Discrete Speech Representation from Self-supervised Learning Model Jiatong Shi Xutai Ma Hirofumi Inaguma Anna Y. Sun Shinji Watanabe 60 7 0 14 Jun 2024
Multi-Modal Retrieval For Large Language Model Based Speech Recognition J. Kolehmainen Aditya Gourav Prashanth Gurunath Shivakumar Yile Gu Ankur Gandhe Ariya Rastrow Grant P. Strimel I. Bulyko 40 4 0 13 Jun 2024
DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding Suwon Shon Kwangyoun Kim Yi-Te Hsu Prashant Sridhar Shinji Watanabe Karen Livescu AuLLM 46 2 0 13 Jun 2024
Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation Se Jin Park Chae Won Kim Hyeongseop Rha Minsu Kim Joanna Hong Jeong Hun Yeo Yong Man Ro CVBM AuLLM 42 6 0 12 Jun 2024
The Interspeech 2024 Challenge on Speech Processing Using Discrete Units Xuankai Chang Jiatong Shi Jinchuan Tian Yuning Wu Yuxun Tang Yihan Wu Shinji Watanabe Yossi Adi Xie Chen Qin Jin 47 15 0 11 Jun 2024
Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing V. Trinh Rosy Southwell Yiwen Guan Xinlu He Zhiyong Wang Jacob Whitehill OffRL 36 2 0 04 Jun 2024
TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation Chenyang Le Yao Qian Dongmei Wang Long Zhou Shujie Liu ... Midia Yousefi Yanmin Qian Jinyu Li Sheng Zhao Michael Zeng 39 3 0 28 May 2024
MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation Yifan Peng Ilia Kulikov Yilin Yang Sravya Popuri Hui Lu Changhan Wang Hongyu Gong 28 4 0 19 Mar 2024
An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesis Yifan Peng Ilia Kulikov Yilin Yang Sravya Popuri Hui Lu Changhan Wang Hongyu Gong 30 1 0 19 Mar 2024
TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages Minsu Kim Jee-weon Jung Hyeongseop Rha Soumi Maiti Siddhant Arora Xuankai Chang Shinji Watanabe Y. Ro 28 7 0 25 Feb 2024
Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations Guan-Ting Lin Cheng-Han Chiang Hung-yi Lee 34 22 0 20 Feb 2024
SpiRit-LM: Interleaved Spoken and Written Language Model Tu Nguyen Benjamin Muller Bokai Yu Marta R. Costa-jussá Maha Elbayad ... Itai Gat Gabriel Synnaeve Juan Pino Benoît Sagot Emmanuel Dupoux AuLLM VLM 51 33 0 08 Feb 2024
SpeechComposer: Unifying Multiple Speech Tasks with Prompt Composition Yihan Wu Soumi Maiti Yifan Peng Wangyou Zhang Chenda Li Yuyue Wang Xihua Wang Shinji Watanabe Ruihua Song 27 3 0 31 Jan 2024
Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation Minsu Kim Jeong Hun Yeo Se Jin Park J. Choi Y. Ro 27 5 0 18 Jan 2024
AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation J. Choi Se Jin Park Minsu Kim Y. Ro 30 12 0 05 Dec 2023
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models Yunfei Chu Jin Xu Xiaohuan Zhou Qian Yang Shiliang Zhang Zhijie Yan Chang Zhou Jingren Zhou AuLLM 42 268 0 14 Nov 2023
UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions Siddhant Arora Hayato Futami Jee-weon Jung Yifan Peng Roshan S. Sharma Yosuke Kashiwagi E. Tsunoo Karen Livescu Shinji Watanabe ELM 27 7 0 04 Oct 2023
UniAudio: An Audio Foundation Model Toward Universal Audio Generation Dongchao Yang Jinchuan Tian Xuejiao Tan Rongjie Huang Songxiang Liu ... Jiang Bian Xixin Wu Zhou Zhao Shinji Watanabe Helen M. Meng CVBM AuLLM 28 114 0 01 Oct 2023
E-Branchformer: Branchformer with Enhanced merging for speech recognition Kwangyoun Kim Felix Wu Yifan Peng Jing Pan Prashant Sridhar Kyu Jeong Han Shinji Watanabe 58 105 0 30 Sep 2022
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing Junyi Ao Rui Wang Long Zhou Chengyi Wang Shuo Ren ... Yu Zhang Zhihua Wei Yao Qian Jinyu Li Furu Wei 118 193 0 14 Oct 2021
Generative Spoken Language Modeling from Raw Audio Kushal Lakhotia Evgeny Kharitonov Wei-Ning Hsu Yossi Adi Adam Polyak ... Tu Nguyen Jade Copet Alexei Baevski A. Mohamed Emmanuel Dupoux AuLLM 191 337 0 01 Feb 2021