SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models

31 August 2023

Xipeng Qiu

Papers citing "SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models"

50 / 55 papers shown

Title
Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play Yemin Shi Yu Shu Siwei Dong Guangyi Liu Jaward Sesay Jingwen Li Zhiting Hu AuLLM VLM 50 0 0 05 May 2025
Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities Xuzhi Zhang Jintao Guo Shanshan Zhao Minghao Fu Lunhao Duan Guo-Hua Wang Qing-Guo Chen Zhao Xu Weihua Luo Kaifu Zhang DiffM 74 0 0 05 May 2025
Deep Audio Watermarks are Shallow: Limitations of Post-Hoc Watermarking Techniques for Speech P. O'Reilly Zeyu Jin Jiaqi Su Bryan Pardo 31 0 0 15 Apr 2025
TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling Liang-Hsuan Tseng Yi-Chang Chen Kuan-Yi Lee Da-shan Shiu Hung-yi Lee AuLLM 56 0 0 09 Apr 2025
UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation Alexander H. Liu Sang-gil Lee Chao-Han Huck Yang Yuan Gong Yu-Chun Wang James Glass Rafael Valle Bryan Catanzaro SSL 52 0 0 02 Mar 2025
From Principles to Applications: A Comprehensive Survey of Discrete Tokenizers in Generation, Comprehension, Recommendation, and Information Retrieval Jian Jia Jingtong Gao Ben Xue Junhao Wang Qingpeng Cai Quan Chen Xiangyu Zhao Peng Jiang Kun Gai OffRL 77 0 0 18 Feb 2025
AudioMiXR: Spatial Audio Object Manipulation with 6DoF for Sound Design in Augmented Reality Brandon Woodard Margarita Geleta Joseph J. LaViola Jr. Andrea Fanelli Rhonda Wilson 57 1 0 05 Feb 2025
SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characters Jianping Jiang Weiye Xiao Zhengyu Lin H. Zhang Tianxiang Ren Yang Gao Zhiqian Lin Zhongang Cai Lei Yang Ziwei Liu 86 3 0 29 Nov 2024
Scaling Transformers for Low-Bitrate High-Quality Speech Coding Julian Parker Anton Smirnov Jordi Pons CJ Carr Zack Zukowski Zach Evans Xubo Liu 77 9 0 29 Nov 2024
MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate Scenarios Xiao-Hang Jiang Yang Ai Rui Zheng Hui-Peng Du Ye-Xin Lu Zhen-Hua Ling 48 2 0 01 Nov 2024
Optimizing Neural Speech Codec for Low-Bitrate Compression via Multi-Scale Encoding Peiji Yang Fengping Wang Yicheng Zhong Huawei Wei Zhisheng Wang 23 0 0 21 Oct 2024
Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant Alan Dao Dinh Bach Vu Huy Hoang Ha AuLLM VLM 70 3 0 20 Oct 2024
DM-Codec: Distilling Multimodal Representations for Speech Tokenization Md Mubtasim Ahasan Md Fahim Tasnim Mohiuddin A K M Mahbubur Rahman Aman Chadha Tariq Iqbal M. A. Amin Md. Mofijul Islam Amin Ahsan Ali 25 0 0 19 Oct 2024
Code Drift: Towards Idempotent Neural Audio Codecs P. O'Reilly Prem Seetharaman Jiaqi Su Zeyu Jin Bryan Pardo 125 0 0 14 Oct 2024
Graded Suspiciousness of Adversarial Texts to Human Shakila Mahjabin Tonni Pedro Faustini Mark Dras AAML 23 0 0 06 Oct 2024
SyllableLM: Learning Coarse Semantic Units for Speech Language Models Alan Baade Puyuan Peng David Harwath 52 3 0 05 Oct 2024
Recent Advances in Speech Language Models: A Survey Wenqian Cui Dianzhi Yu Xiaoqi Jiao Ziqiao Meng Guangyan Zhang Qichao Wang Yiwen Guo Irwin King AuLLM 59 14 0 01 Oct 2024
Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Models Wenrui Liu Zhifang Guo Jin Xu Yuanjun Lv Yunfei Chu Zhou Zhao Junyang Lin 53 1 0 28 Sep 2024
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions Kai Chen Yunhao Gou Runhui Huang Zhili Liu Daxin Tan ... Qun Liu Jun Yao Lu Hou Hang Xu Hang Xu AuLLM MLLM VLM 82 21 0 26 Sep 2024
Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM Robin Shing-Hei Yuen Timothy Tin-Long Tse Jian Zhu AuLLM 33 3 0 25 Sep 2024
StyleFusion TTS: Multimodal Style-control and Enhanced Feature Fusion for Zero-shot Text-to-speech Synthesis Zhiyong Chen Xinnuo Li Zhiqi Ai Shugong Xu DiffM 36 1 0 24 Sep 2024
Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models Haibin Wu Xuanjun Chen Yi-Cheng Lin Kaiwei Chang Jiawei Du ... Yi-Chiao Wu Xu Tan James Glass Shinji Watanabe Hung-yi Lee 34 6 0 21 Sep 2024
Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility Xiaoyu Liu Xu Li Joan Serrà Santiago Pascual 31 3 0 14 Sep 2024
Text-To-Speech Synthesis In The Wild Jee-weon Jung Wangyou Zhang Soumi Maiti Yihan Wu Xin Wang ... Hye-jin Shim Nicholas W. D. Evans Joon Son Chung Shinnosuke Takamichi Shinji Watanabe 38 1 0 13 Sep 2024
LAST: Language Model Aware Speech Tokenization A. Turetzky Yossi Adi 34 2 0 05 Sep 2024
SoCodec: A Semantic-Ordered Multi-Stream Speech Codec for Efficient Language Model Based Text-to-Speech Synthesis Haohan Guo Fenglong Xie Kun Xie Dongchao Yang Dake Guo Xixin Wu Helen Meng 34 4 0 02 Sep 2024
MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer Yuancheng Wang Haoyue Zhan Liwei Liu Ruihong Zeng Haotian Guo Jiachen Zheng Qiang Zhang Shunsi Zhang Shunsi Zhang Zhizheng Wu 36 39 0 01 Sep 2024
Progressive Residual Extraction based Pre-training for Speech Representation Learning Tianrui Wang Jin Li Ziyang Ma Rui Cao Xie Chen ... Meng Ge Xiaobao Wang Yuguang Wang Jianwu Dang Nyima Tashi SSL 43 0 0 31 Aug 2024
SSDM: Scalable Speech Dysfluency Modeling Jiachen Lian Xuanru Zhou Z. Ezzes Jet M J Vonk Brittany Morin D. Baquirin Zachary Mille M. G. Tempini Gopala Anumanchipalli AuLLM 32 1 0 29 Aug 2024
WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling Shengpeng Ji Ziyue Jiang Xize Cheng Yifu Chen Minghui Fang ... Rongjie Huang Yidi Jiang Qian Chen Zhou Zhao Zhou Zhao VLM 57 33 0 29 Aug 2024
dMel: Speech Tokenization made Simple Richard He Bai Tatiana Likhomanenko Ruixiang Zhang Zijin Gu Zakaria Aldeneh Navdeep Jaitly 35 4 0 22 Jul 2024
Overview of Speaker Modeling and Its Applications: From the Lens of Deep Speaker Representation Learning Shuai Wang Zheng-Shou Chen Kong Aik Lee Yan-min Qian Haizhou Li 39 4 0 21 Jul 2024
Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations Kunal Dhawan Nithin Rao Koluguri Ante Jukić Ryan Langman Jagadeesh Balam Boris Ginsburg 47 1 0 03 Jul 2024
Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimization Yuchen Hu Chen Chen Siyin Wang Eng Siong Chng C. Zhang 43 3 0 02 Jul 2024
DASB -- Discrete Audio and Speech Benchmark Pooneh Mousavi Luca Della Libera J. Duret Artem Ploujnikov Cem Subakan Mirco Ravanelli 35 12 0 20 Jun 2024
Articulatory Encodec: Coding Speech through Vocal Tract Kinematics Cheol Jun Cho Peter Wu Tejas S. Prabhune Dhruv Agarwal Gopala K. Anumanchipalli 36 1 0 18 Jun 2024
UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learner Dongchao Yang Haohan Guo Yuanyuan Wang Rongjie Huang Xiang Li Xu Tan Xixin Wu Helen Meng AuLLM 44 15 0 14 Jun 2024
Autoregressive Diffusion Transformer for Text-to-Speech Synthesis Zhijun Liu Shuai Wang Sho Inoue Qibing Bai Haizhou Li DiffM 47 15 0 08 Jun 2024
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers Sanyuan Chen Shujie Liu Long Zhou Yanqing Liu Xu Tan Jinyu Li Sheng Zhao Yao Qian Furu Wei VLM 47 65 0 08 Jun 2024
Neural Codec-based Adversarial Sample Detection for Speaker Verification Xuanjun Chen Jiawei Du Haibin Wu Jyh-Shing Roger Jang Hung-yi Lee 34 2 0 07 Jun 2024
Addressing Index Collapse of Large-Codebook Speech Tokenizer with Dual-Decoding Product-Quantized Variational Auto-Encoder Haohan Guo Fenglong Xie Dongchao Yang Hui Lu Xixin Wu Helen Meng 57 6 0 05 Jun 2024
ControlSpeech: Towards Simultaneous Zero-shot Speaker Cloning and Zero-shot Language Style Control With Decoupled Codec Shengpeng Ji Jia-li Zuo Minghui Fang Siqi Zheng Qian Chen ... Ziyue Jiang Hai Huang Xize Cheng Rongjie Huang Zhou Zhao 52 8 0 03 Jun 2024
Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback Chen Chen Yuchen Hu Wen Wu Helin Wang Chng Eng Siong Chao Zhang 40 10 0 02 Jun 2024
SpeechAlign: Aligning Speech Generation to Human Preferences Dong Zhang Zhaowei Li Shimin Li Xin Zhang Pengyu Wang Yaqian Zhou Xipeng Qiu ALM AuLLM 32 15 0 08 Apr 2024
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models Zeqian Ju Yuancheng Wang Kai Shen Xu Tan Detai Xin ... Shikun Zhang Jiang Bian Lei He Jinyu Li Sheng Zhao DiffM 46 144 0 05 Mar 2024
Towards audio language modeling -- an overview Haibin Wu Xuanjun Chen Yi-Cheng Lin Kai-Wei Chang Ho-Lam Chung Alexander H. Liu Hung-yi Lee AuLLM 30 28 0 20 Feb 2024
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling Jun Zhan Junqi Dai Jiasheng Ye Yunhua Zhou Dong Zhang ... Jie Fu Tao Gui Tianxiang Sun Yugang Jiang Xipeng Qiu MLLM 27 114 0 19 Feb 2024
APCodec: A Neural Audio Codec with Parallel Amplitude and Phase Spectrum Encoding and Decoding Yang Ai Xiao-Hang Jiang Ye-Xin Lu Hui-Peng Du Zhenhua Ling 21 20 0 16 Feb 2024
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data Mateusz Lajszczak Guillermo Cámbara Yang Li Fatih Beyhan Arent van Korlaar ... Bartosz Putrycz Soledad López Gambino Kayeon Yoo Elena Sokolova Thomas Drugman LM&MA 38 72 0 12 Feb 2024
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models Yunfei Chu Jin Xu Xiaohuan Zhou Qian Yang Shiliang Zhang Zhijie Yan Chang Zhou Jingren Zhou AuLLM 42 268 0 14 Nov 2023