v1v2 (latest)

SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

18 May 2023

Xipeng Qiu

Papers citing "SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities"

40 / 40 papers shown

Title
MFA-KWS: Effective Keyword Spotting with Multi-head Frame-asynchronous Decoding Yu Xi Haoyu Li Xiaoyu Gu Yidi Jiang Kai Yu 54 1 0 26 May 2025
Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models Chi-Yuan Hsiao Ke-Han Lu Kai-Wei Chang Chih-Kai Yang Wei-Chih Chen Hung-yi Lee CLL MoMe 189 0 0 23 May 2025
LLM as Effective Streaming Processor: Bridging Streaming-Batch Mismatches with Group Position Encoding Junlong Tong Jinlan Fu Zixuan Lin Yingqi Fan Anhao Zhao Hui Su Xiaoyu Shen 86 0 0 22 May 2025
AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Models Guangke Chen Fu Song Zhe Zhao Xiaojun Jia Yang Liu Yanchen Qiao Weizhe Zhang AuLLM AAML 78 1 0 20 May 2025
EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting Guanrou Yang Chen Yang Qian Chen Ziyang Ma Wenxi Chen ... Fan Yu Zhihao Du Zhifu Gao Shiliang Zhang Xie Chen AuLLM 160 1 0 17 Apr 2025
Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations Xue Jiang Xiulian Peng Yuan Zhang Yan Lu SSL 136 1 0 15 Mar 2025
Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision Che Liu Yingji Zhang D. Zhang Weijie Zhang Chenggong Gong ... André Freitas Qifan Wang Z. Xu Rongjuncheng Zhang Yong Dai AuLLM 219 2 0 26 Feb 2025
FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systems Borui Liao Yulong Xu Jiao Ou Kaiyuan Yang Weihua Jian Pengfei Wan Di Zhang AuLLM 133 0 0 19 Feb 2025
DuplexMamba: Enhancing Real-time Speech Conversations with Duplex and Streaming Capabilities Xiangyu Lu Wang Xu Haoyu Wang Hongyun Zhou Haiyan Zhao Conghui Zhu Tiejun Zhao M. Yang Mamba AuLLM 107 0 0 16 Feb 2025
SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words Junyi Ao Yuancheng Wang Xiaohai Tian Dekun Chen Jing Zhang Lu Lu Yansen Wang Haizhou Li Zhikai Wu AuLLM 150 24 0 17 Jan 2025
A Comprehensive Survey of Foundation Models in Medicine Wasif Khan Seowung Leem Kyle B. See Joshua K. Wong Shaoting Zhang R. Fang AI4CE LM&MA VLM 276 26 0 17 Jan 2025
OneLLM: One Framework to Align All Modalities with Language Jiaming Han Kaixiong Gong Yiyuan Zhang Jiaqi Wang Kaipeng Zhang Dahua Lin Yu Qiao Peng Gao Xiangyu Yue MLLM 226 132 0 10 Jan 2025
"Yeah Right!" -- Do LLMs Exhibit Multimodal Feature Transfer? Benjamin Z. Reichman Kartik Talamadupula 86 0 0 07 Jan 2025
Instruction-Guided Scene Text Recognition Yongkun Du Z. Chen Yuchen Su Caiyan Jia Yu-Gang Jiang 182 3 0 03 Jan 2025
AMPS: ASR with Multimodal Paraphrase Supervision Amruta Parulekar Abhishek Gupta Sameep Chattopadhyay Preethi Jyothi 101 0 0 27 Nov 2024
Spider: Any-to-Many Multimodal LLM Jinxiang Lai Jie Zhang Jun Liu Jian Li Xiaocheng Lu Song Guo MLLM 142 2 0 14 Nov 2024
Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback Guan-Ting Lin Prashanth Gurunath Shivakumar Aditya Gourav Yile Gu Ankur Gandhe Hung-yi Lee I. Bulyko 107 9 0 04 Nov 2024
SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Models Yue Zhang Zhiyang Xu Ying Shen Parisa Kordjamshidi Lifu Huang 114 8 0 04 Oct 2024
Recent Advances in Speech Language Models: A Survey Wenqian Cui Dianzhi Yu Xiaoqi Jiao Ziqiao Meng Guangyan Zhang Qichao Wang Yiwen Guo Irwin King AuLLM 159 25 0 01 Oct 2024
SSR: Alignment-Aware Modality Connector for Speech Language Models Weiting Tan Hirofumi Inaguma Ning Dong Paden Tomasello Xutai Ma 110 6 0 30 Sep 2024
MIO: A Foundation Model on Multimodal Tokens Zekun Wang King Zhu Chunpu Xu Wangchunshu Zhou Jiaheng Liu ... Yuanxing Zhang Ge Zhang Ke Xu Jie Fu Wenhao Huang MLLM AuLLM 130 12 0 26 Sep 2024
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions Kai Chen Yunhao Gou Runhui Huang Zhili Liu Daxin Tan ... Qun Liu Jun Yao Lu Hou Hang Xu Hang Xu AuLLM MLLM VLM 147 29 0 26 Sep 2024
Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models Potsawee Manakul Guangzhi Sun Warit Sirichotedumrong Kasima Tharnpipitchai Kunat Pipatanakul AuLLM 86 7 0 17 Sep 2024
WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling Shengpeng Ji Ziyue Jiang Xize Cheng Yifu Chen Minghui Fang ... Rongjie Huang Yidi Jiang Qian Chen Zhou Zhao Zhou Zhao VLM 124 44 0 29 Aug 2024
dMel: Speech Tokenization made Simple Richard He Bai Tatiana Likhomanenko Ruixiang Zhang Zijin Gu Zakaria Aldeneh Navdeep Jaitly 96 6 0 22 Jul 2024
AudioBench: A Universal Benchmark for Audio Large Language Models Bin Wang Xunlong Zou Geyu Lin Siyang Song Zhuohan Liu Wenyu Zhang Zhengyuan Liu AiTi Aw Nancy F. Chen AuLLM ELM LM&MA 143 35 0 23 Jun 2024
Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Models Xi Li Ruofan Mao Yusen Zhang Renze Lou Chen Wu Jiaqi Wang LRM AAML 72 14 0 10 Jun 2024
SpeechVerse: A Large-scale Generalizable Audio Language Model Nilaksh Das Saket Dingliwal S. Ronanki Rohit Paturi David Huang ... Monica Sunkara S. Srinivasan Kyu J. Han Katrin Kirchhoff Katrin Kirchhoff 75 43 0 14 May 2024
Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs Yu Xia Rui Wang Xu Liu Mingyan Li Tong Yu Xiang Chen Julian McAuley Shuai Li LRM 98 22 0 24 Apr 2024
Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning Jiasheng Ye Zaixiang Zheng Yu Bao Lihua Qian Quanquan Gu DiffM 135 19 0 23 Aug 2023
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face Yongliang Shen Kaitao Song Xu Tan Dongsheng Li Weiming Lu Yueting Zhuang MLLM 122 908 0 30 Mar 2023
AudioLM: a Language Modeling Approach to Audio Generation Zalan Borsos Raphaël Marinier Damien Vincent Eugene Kharitonov Olivier Pietquin ... Dominik Roblek O. Teboul David Grangier Marco Tagliasacchi Neil Zeghidour AuLLM 161 615 0 07 Sep 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 522 6,293 0 05 Apr 2022
A General Language Assistant as a Laboratory for Alignment Amanda Askell Yuntao Bai Anna Chen Dawn Drain Deep Ganguli ... Tom B. Brown Jack Clark Sam McCandlish C. Olah Jared Kaplan ALM 120 789 0 01 Dec 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 490 10,496 0 17 Jun 2021
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units Wei-Ning Hsu Benjamin Bolte Yao-Hung Hubert Tsai Kushal Lakhotia Ruslan Salakhutdinov Abdel-rahman Mohamed SSL 182 2,993 0 14 Jun 2021
GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio Guoguo Chen Shuzhou Chai Guan-Bo Wang Jiayu Du Weiqiang Zhang ... Xuchen Yao Yongqing Wang Yujun Wang Zhao You Zhiyong Yan 116 383 0 13 Jun 2021
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations Alexei Baevski Henry Zhou Abdel-rahman Mohamed Michael Auli SSL 297 5,837 0 20 Jun 2020
Libri-Light: A Benchmark for ASR with Limited or No Supervision Jacob Kahn M. Rivière Weiyi Zheng Evgeny Kharitonov Qiantong Xu ... Tatiana Likhomanenko Gabriel Synnaeve Armand Joulin Abdel-rahman Mohamed Emmanuel Dupoux AuLLM 75 673 0 17 Dec 2019
Common Voice: A Massively-Multilingual Speech Corpus Rosana Ardila Megan Branson Kelly Davis Michael Henretty M. Kohler Josh Meyer Reuben Morais Lindsay Saunders Francis M. Tyers Gregor Weber VLM 91 1,614 0 13 Dec 2019