SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

18 May 2023

Xipeng Qiu

Papers citing "SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities"

50 / 226 papers shown

Title
$$\mathcal{A}LLM4ADD$: Unlocking the Capabilities of Audio Large Language Models for Audio Deepfake Detection$ $\mathcal{A}LLM4ADD$ : Unlocking the Capabilities of Audio Large Language Models for Audio Deepfake Detection Hao Gu Jiangyan Yi Chenglong Wang Jianhua Tao Zheng Lian Jiayi He Yong Ren Yujie Chen Zhengqi Wen 12 0 0 16 May 2025
WavReward: Spoken Dialogue Models With Generalist Reward Evaluators Shengpeng Ji Tianle Liang Yongqian Li Jialong Zuo Minghui Fang ... Xize Cheng Siqi Zheng Jin Xu Junyang Lin Zhou Zhao AuLLM ALM 33 0 0 14 May 2025
Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding Dianwen Ng Kun Zhou Yi-Wen Chao Zhiwei Xiong B. Ma E. Chng 33 0 0 12 May 2025
LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis Qingkai Fang Yan Zhou Shoutao Guo Shaolei Zhang Yang Feng AuLLM 56 1 0 05 May 2025
Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play Yemin Shi Yu Shu Siwei Dong Guangyi Liu Jaward Sesay Jingwen Li Zhiting Hu AuLLM VLM 50 0 0 05 May 2025
Symbolic Representation for Any-to-Any Generative Tasks Jianfei Chen Xiaoye Zhu Yunhong Wang Tianyang Liu Xinhui Chen ... Yifei Ke Jiaheng Liu Yiwen Yuan Julian McAuley Li Li DiffM 38 0 0 24 Apr 2025
A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms Chengkai Huang Hongtao Huang Tong Yu Kaige Xie Junda Wu Shuai Zhang Julian McAuley Dietmar Jannach Lina Yao LRM AI4CE 24 0 0 23 Apr 2025
SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation Keqi Deng Wenxi Chen Xie Chen P. Woodland 43 0 0 22 Apr 2025
EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting Guanrou Yang Chen Yang Qian Chen Ziyang Ma Wenxi Chen ... Fan Yu Zhihao Du Zhifu Gao Shiliang Zhang Xie Chen AuLLM 57 0 0 17 Apr 2025
Voice Interaction With Conversational AI Could Facilitate Thoughtful Reflection and Substantive Revision in Writing Jiho Kim Philippe Laban Xiang Ánthony' Chen Kenneth C. Arnold 35 0 0 11 Apr 2025
On The Landscape of Spoken Language Models: A Comprehensive Survey Siddhant Arora Kai-Wei Chang Chung-Ming Chien Yifan Peng Haibin Wu Yossi Adi Emmanuel Dupoux Hung-yi Lee Karen Livescu Shinji Watanabe 52 2 0 11 Apr 2025
Scaling Analysis of Interleaved Speech-Text Language Models Gallil Maimon Michael Hassid Amit Roth Yossi Adi AuLLM 43 0 0 03 Apr 2025
SViQA: A Unified Speech-Vision Multimodal Model for Textless Visual Question Answering Bingxin Li 30 0 0 01 Apr 2025
FinAudio: A Benchmark for Audio Large Language Models in Financial Applications Yupeng Cao Haohang Li Yangyang Yu Shashidhar Reddy Javaji Yueru He ... Xiao-Yang Liu K. P. Subbalakshmi Meikang Qiu Sophia Ananiadou J. Nie AuLLM 71 0 0 26 Mar 2025
Dolphin: A Large-Scale Automatic Speech Recognition Model for Eastern Languages Yangyang Meng Jinpeng Li Guodong Lin Yu Pu G. Wang Hu Du Zhiming Shao Yukai Huang Ke Li Wei-Qiang Zhang ObjD 99 0 0 26 Mar 2025
CountLLM: Towards Generalizable Repetitive Action Counting via Large Language Model Ziyu Yao Xuxin Cheng Zhiqi Huang Lei Li 59 0 0 22 Mar 2025
Multi-modal Time Series Analysis: A Tutorial and Survey Yushan Jiang Kanghui Ning Zijie Pan Xuyang Shen Jingchao Ni Wenchao Yu Anderson Schneider Haifeng Chen Yuriy Nevmyvaka Dongjin Song AI4TS 155 0 0 17 Mar 2025
Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey Yixuan Wang Shengqiong Wu Yuyao Zhang William Yang Wang Ziwei Liu Jiebo Luo Hao Fei LRM 92 8 0 16 Mar 2025
Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations Xue Jiang Xiulian Peng Yuan Zhang Yan-Heng Lu SSL 83 0 0 15 Mar 2025
From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM Kshitij Ambilduke Ben Peters Sonal Sannigrahi Anil Keshwani Tsz Kin Lam Bruno Martins Marcely Zanon Boito André F. T. Martins 52 0 0 13 Mar 2025
Adaptive Inner Speech-Text Alignment for LLM-based Speech Translation Henglyu Liu Andong Chen Kehai Chen X. Bai M. Zhong Yuan Qiu Min Zhang 42 0 0 13 Mar 2025
ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems Siddhant Arora Yifan Peng Jiatong Shi Jinchuan Tian William Chen ... Yosuke Kashiwagi E. Tsunoo Shuichiro Shimizu Vaibhav Srivastav Shinji Watanabe 42 0 0 11 Mar 2025
3D Point Cloud Generation via Autoregressive Up-sampling Ziqiao Meng Qichao Wang Zhipeng Zhou Irwin King Peilin Zhao 3DPC 61 0 0 11 Mar 2025
Text-Speech Language Models with Improved Cross-Modal Transfer by Aligning Abstraction Levels Santiago Cuervo Adel Moumen Yanis Labrak Sameer Khurana Antoine Laurent Mickael Rouvier R. Marxer 77 1 0 08 Mar 2025
S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following with Paralinguistic Information Feng Jiang Zhiyu Lin Fan Bu Yuhao Du Benyou Wang Hao Li AuLLM ELM 98 0 0 07 Mar 2025
LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM Shri Kiran Srinivasan Mohammed Irfan Kurpath Sahal Shaji Mullappilly Jean Lahoud Fahad A Khan Rao Muhammad Anwer Salman Khan Hisham Cholakkal AuLLM 150 0 0 06 Mar 2025
Enhancing Spoken Discourse Modeling in Language Models Using Gestural Cues Varsha Suresh Muhammad Hamza Mughal Christian Theobalt Vera Demberg 53 0 0 05 Mar 2025
InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-training Dingdong Wang Jin Xu Ruihang Chu Zhifang Guo Xinyu Wang Jincenzi Wu Dongchao Yang Shengpeng Ji Junyang Lin AuLLM 83 0 0 04 Mar 2025
Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models Zhifei Xie Mingbao Lin Ziqiang Liu Pengcheng Wu Shuicheng Yan Chunyan Miao AuLLM OffRL LRM 79 7 0 04 Mar 2025
UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation Alexander H. Liu Sang-gil Lee Chao-Han Huck Yang Yuan Gong Yu-Chun Wang James Glass Rafael Valle Bryan Catanzaro SSL 52 0 0 02 Mar 2025
LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement Boyi Kang Xinfa Zhu Zihan Zhang Zhen Ye Mingshuai Liu ... Jun Chen Longshuai Xiao Chao Weng Wei Xue Lei Xie AuLLM 55 3 0 01 Mar 2025
PodAgent: A Comprehensive Framework for Podcast Generation Yujia Xiao Lei He Haohan Guo Fenglong Xie Tan Lee 147 0 0 01 Mar 2025
Does Your Voice Assistant Remember? Analyzing Conversational Context Recall and Utilization in Voice Interaction Models Heeseung Kim Che Hyun Lee S. Park Jiheum Yeom Nohil Park Sangwon Yu Sungroh Yoon 71 0 0 27 Feb 2025
Nexus-O: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision Che Liu Yingji Zhang D. Zhang Weijie Zhang Chenggong Gong ... André Freitas Qifan Wang Z. Xu Rongjuncheng Zhang Yong Dai AuLLM 76 0 0 26 Feb 2025
Mind the Gap! Static and Interactive Evaluations of Large Audio Models Minzhi Li William B. Held Michael Joseph Ryan Kunat Pipatanakul Potsawee Manakul Hao Zhu Diyi Yang AuLLM ALM 58 0 0 21 Feb 2025
FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systems Borui Liao Yulong Xu Jiao Ou Kaiyuan Yang Weihua Jian Pengfei Wan Di Zhang AuLLM 62 0 0 20 Feb 2025
DuplexMamba: Enhancing Real-time Speech Conversations with Duplex and Streaming Capabilities Xiangyu Lu Wang Xu Haoyu Wang Hongyun Zhou Haiyan Zhao Conghui Zhu T. Zhao M. Yang Mamba AuLLM 66 0 0 16 Feb 2025
From No to Know: Taxonomy, Challenges, and Opportunities for Negation Understanding in Multimodal Foundation Models Mayank Vatsa Aparna Bharati S. Mittal Richa Singh 58 0 0 10 Feb 2025
Audio-Language Models for Audio-Centric Tasks: A survey Yi Su Jisheng Bai Qisheng Xu Kele Xu Yong Dou AuLLM 99 2 0 28 Jan 2025
A Comprehensive Survey of Foundation Models in Medicine Wasif Khan Seowung Leem Kyle B. See Joshua K. Wong Shaoting Zhang R. Fang AI4CE LM&MA VLM 105 18 0 17 Jan 2025
SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words Junyi Ao Yuancheng Wang Xiaohai Tian Dekun Chen Jingyang Zhang Lu Lu Yixuan Wang Haizhou Li Zhikai Wu AuLLM 87 17 0 17 Jan 2025
OneLLM: One Framework to Align All Modalities with Language Jiaming Han Kaixiong Gong Yiyuan Zhang Jiaqi Wang Kaipeng Zhang Dahua Lin Yu Qiao Peng Gao Xiangyu Yue MLLM 104 109 0 10 Jan 2025
"Yeah Right!" -- Do LLMs Exhibit Multimodal Feature Transfer? Benjamin Z. Reichman Kartik Talamadupula 41 0 0 07 Jan 2025
Instruction-Guided Scene Text Recognition Yongkun Du Z. Chen Yuchen Su Caiyan Jia Yu-Gang Jiang 75 3 0 03 Jan 2025
OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios Xize Cheng Dongjie Fu Xiaoda Yang Minghui Fang Ruofan Hu ... Rongjie Huang Linjun Li Yu Chen Tao Jin Zhou Zhao 46 1 0 03 Jan 2025
AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues Se Jin Park Yeonju Kim Hyeongseop Rha Bella Godiva Y. Ro 36 1 0 23 Dec 2024
SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization Tan-Hanh Pham Hoang-Nam Le Phu-Vinh Nguyen Chris Ngo Truong Son-Hy AuLLM LRM 81 1 0 21 Dec 2024
The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion Changan Chen Juze Zhang S. K. Lakshmikanth Yusu Fang Ruizhi Shao Gordon Wetzstein L. Fei-Fei Ehsan Adeli VGen 82 3 0 13 Dec 2024
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions Pan Zhang Xiaoyi Dong Yuhang Cao Yuhang Zang Rui Qian ... Xiaotian Zhang K. Chen Yu Qiao Dahua Lin Jiaqi Wang KELM 84 12 0 12 Dec 2024
Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition Zhisheng Zhong Chengyao Wang Yuqi Liu Senqiao Yang Longxiang Tang ... Shaozuo Yu Sitong Wu Eric Lo Shu-Lin Liu Jiaya Jia AuLLM 105 6 0 12 Dec 2024