Improving Audio Captioning Models with Fine-grained Audio Features, Text Embedding Supervision, and LLM Mix-up Augmentation

29 September 2023

Papers citing "Improving Audio Captioning Models with Fine-grained Audio Features, Text Embedding Supervision, and LLM Mix-up Augmentation"

30 / 30 papers shown

Title
JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models Zifan Peng Yule Liu Zhen Sun Mingchen Li Zeren Luo ... Xinlei He Xuechao Wang Yingjie Xue Shengmin Xu Xinyi Huang AuLLM AAML 75 1 0 23 May 2025
Audio-Language Datasets of Scenes and Events: A Survey Gijs Wijngaard Elia Formisano Michele Esposito M. Dumontier 114 2 0 10 Jan 2025
Efficient Audio Captioning Transformer with Patchout and Text Guidance Thodoris Kouzelis Grigoris Bastas Athanasios Katsamanis Alexandros Potamianos ViT 69 6 0 06 Apr 2023
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research Xinhao Mei Chutong Meng Haohe Liu Qiuqiang Kong Tom Ko Chengqi Zhao Mark D. Plumbley Yuexian Zou Wenwu Wang 87 209 0 30 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 1.3K 13,100 0 27 Feb 2023
One Embedder, Any Task: Instruction-Finetuned Text Embeddings Hongjin Su Weijia Shi Jungo Kasai Yizhong Wang Yushi Hu Mari Ostendorf Wen-tau Yih Noah A. Smith Luke Zettlemoyer Tao Yu 85 296 0 19 Dec 2022
BEATs: Audio Pre-Training with Acoustic Tokenizers Sanyuan Chen Yu-Huan Wu Chengyi Wang Shujie Liu Daniel C. Tompkins Zhuo Chen Furu Wei 87 285 0 18 Dec 2022
MTEB: Massive Text Embedding Benchmark Niklas Muennighoff Nouamane Tazi L. Magne Nils Reimers 447 395 0 13 Oct 2022
Masked Autoencoders that Listen Po-Yao (Bernie) Huang Hu Xu Juncheng Billy Li Alexei Baevski Michael Auli Wojciech Galuba Florian Metze Christoph Feichtenhofer 68 280 0 13 Jul 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 767 12,835 0 04 Mar 2022
HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection Ke Chen Xingjian Du Bilei Zhu Zejun Ma Taylor Berg-Kirkpatrick Shlomo Dubnov ViT 139 269 0 02 Feb 2022
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing Sanyuan Chen Chengyi Wang Zhengyang Chen Yu-Huan Wu Shujie Liu ... Yao Qian Jian Wu Micheal Zeng Xiangzhan Yu Furu Wei SSL 206 1,846 0 26 Oct 2021
Can Audio Captions Be Evaluated with Image Caption Metrics? Zelin Zhou Zhiling Zhang Xuenan Xu Zeyu Xie Mengyue Wu Kenny Q. Zhu 47 46 0 10 Oct 2021
CIDEr-R: Robust Consensus-based Image Description Evaluation G. O. D. Santos Esther Luna Colombini Sandra Avila 54 30 0 28 Sep 2021
An Encoder-Decoder Based Audio Captioning System With Transfer and Reinforcement Learning Xinhao Mei Qiushi Huang Xubo Liu Gengyun Chen Jingqian Wu ... Tom Ko H. Tang Xingkun Shao Mark D. Plumbley Wenwu Wang 47 53 0 05 Aug 2021
CLIPScore: A Reference-free Evaluation Metric for Image Captioning Jack Hessel Ari Holtzman Maxwell Forbes Ronan Le Bras Yejin Choi CLIP 117 1,545 0 18 Apr 2021
PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation Yuan Gong Yu-An Chung James R. Glass VLM 154 147 0 02 Feb 2021
Conformer: Convolution-augmented Transformer for Speech Recognition Anmol Gulati James Qin Chung-Cheng Chiu Niki Parmar Yu Zhang ... Wei Han Shibo Wang Zhengdong Zhang Yonghui Wu Ruoming Pang 210 3,119 0 16 May 2020
A Simple Framework for Contrastive Learning of Visual Representations Ting-Li Chen Simon Kornblith Mohammad Norouzi Geoffrey E. Hinton SSL 333 18,721 0 13 Feb 2020
PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition Qiuqiang Kong Yin Cao Turab Iqbal Yuxuan Wang Wenwu Wang Mark D. Plumbley VLM SSL 174 1,074 0 21 Dec 2019
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension M. Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdel-rahman Mohamed Omer Levy Veselin Stoyanov Luke Zettlemoyer AIMat VLM 219 10,792 0 29 Oct 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 377 20,053 0 23 Oct 2019
Clotho: An Audio Captioning Dataset Konstantinos Drossos Samuel Lipping Tuomas Virtanen 87 388 0 21 Oct 2019
The Curious Case of Neural Text Degeneration Ari Holtzman Jan Buys Li Du Maxwell Forbes Yejin Choi 170 3,160 0 22 Apr 2019
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition Daniel S. Park William Chan Yu Zhang Chung-Cheng Chiu Barret Zoph E. D. Cubuk Quoc V. Le VLM 162 3,451 0 18 Apr 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.6K 94,511 0 11 Oct 2018
Representation Learning with Contrastive Predictive Coding Aaron van den Oord Yazhe Li Oriol Vinyals DRL SSL 292 10,253 0 10 Jul 2018
mixup: Beyond Empirical Risk Minimization Hongyi Zhang Moustapha Cissé Yann N. Dauphin David Lopez-Paz NoLa 271 9,743 0 25 Oct 2017
Automated Audio Captioning with Recurrent Neural Networks Konstantinos Drossos Sharath Adavanne Tuomas Virtanen 60 129 0 30 Jun 2017
Self-critical Sequence Training for Image Captioning Steven J. Rennie E. Marcheret Youssef Mroueh Jerret Ross Vaibhava Goel 105 1,886 0 02 Dec 2016