HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection

2 February 2022

Ke Chen

Xingjian Du

Bilei Zhu

Zejun Ma

Taylor Berg-Kirkpatrick

Shlomo Dubnov

ViT

ArXiv PDF HTML

Papers citing "HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection"

50 / 53 papers shown

Title
TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining Paul Primus Florian Schmid Gerhard Widmer CLIP AI4TS VLM 33 0 0 12 May 2025
FLAM: Frame-Wise Language-Audio Modeling Yusong Wu Christos Tsirigotis Ke Chen Cheng-Zhi Anna Huang Aaron C. Courville Oriol Nieto Prem Seetharaman Justin Salamon 50 0 0 08 May 2025
Learning Musical Representations for Music Performance Question Answering Xingjian Diao Chunhui Zhang Tingxuan Wu Ming Cheng Z. Ouyang Weiyi Wu Jiang Gui 70 7 0 10 Feb 2025
Hybrid Losses for Hierarchical Embedding Learning Haokun Tian Stefan Lattner Brian McFee Charalampos Saitis 47 0 0 22 Jan 2025
Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach Rory Young Nicolas Pugeault AAML 59 0 0 14 Oct 2024
Semi-intrusive audio evaluation: Casting non-intrusive assessment as a multi-modal text prediction task Jozef Coldenhoff Milos Cernak 36 0 0 21 Sep 2024
Effective Pre-Training of Audio Transformers for Sound Event Detection Florian Schmid T. Morocutti Francesco Foscarin Jan Schluter Paul Primus Gerhard Widmer ViT 28 2 0 14 Sep 2024
Generating Sample-Based Musical Instruments Using Neural Audio Codec Language Models S. Nercessian Johannes Imort Ninon Devis Frederik Blang 38 1 0 22 Jul 2024
FastAST: Accelerating Audio Spectrogram Transformer via Token Merging and Cross-Model Knowledge Distillation Swarup Ranjan Behera Abhishek Dhiman Karthik Gowda Aalekhya Satya Narayani 21 1 0 11 Jun 2024
Bridging Language Gaps in Audio-Text Retrieval Zhiyong Yan Heinrich Dinkel Yongqing Wang Jizhong Liu Junbo Zhang Yujun Wang Bin Wang VLM 39 4 0 11 Jun 2024
Audio-based Step-count Estimation for Running -- Windowing and Neural Network Baselines Philipp Wagner Andreas Triantafyllopoulos Alexander Gebhard Björn Schuller 37 0 0 10 Jun 2024
Multi-Microphone Speech Emotion Recognition using the Hierarchical Token-semantic Audio Transformer Architecture Ohad Cohen G. Hazan Sharon Gannot 34 1 0 05 Jun 2024
HAFFormer: A Hierarchical Attention-Free Framework for Alzheimer's Disease Detection From Spontaneous Speech Zhongren Dong Zixing Zhang Weixiang Xu Jing Han Jianjun Ou Björn W. Schuller 40 1 0 07 May 2024
AudioRepInceptionNeXt: A lightweight single-stream architecture for efficient audio recognition Kin Wai Lau Yasar Abbas Ur Rehman L. Po 44 1 0 21 Apr 2024
Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models David Kurzendörfer Otniel-Bogdan Mercea A. Sophia Koepke Zeynep Akata VLM CLIP 33 2 0 09 Apr 2024
Correlation of Fréchet Audio Distance With Human Perception of Environmental Audio Is Embedding Dependant Modan Tailleur Junwon Lee Mathieu Lagrange Keunwoo Choi Laurie M. Heller Keisuke Imoto Yuki Okamoto 30 10 0 26 Mar 2024
Multiscale Matching Driven by Cross-Modal Similarity Consistency for Audio-Text Retrieval Qian Wang Jia-Chen Gu Zhen-Hua Ling 35 2 0 15 Mar 2024
Audio-Visual LLM for Video Understanding Fangxun Shu Lei Zhang Hao Jiang Cihang Xie VLM MLLM 27 38 0 11 Dec 2023
Audio Prompt Tuning for Universal Sound Separation Yuzhuo Liu Xubo Liu Yan Zhao Yuanyuan Wang Rui Xia Pingchuan Tain Yuxuan Wang VLM 39 5 0 30 Nov 2023
A-JEPA: Joint-Embedding Predictive Architecture Can Listen Zhengcong Fei Mingyuan Fan Junshi Huang 25 17 0 27 Nov 2023
InstrumentGen: Generating Sample-Based Musical Instruments From Text S. Nercessian Johannes Imort 27 2 0 07 Nov 2023
OmniVec: Learning robust representations with cross modal sharing Siddharth Srivastava Gaurav Sharma SSL 27 64 0 07 Nov 2023
CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models Sreyan Ghosh Ashish Seth Sonal Kumar Utkarsh Tyagi Chandra Kiran Reddy Evuru S. Ramaneswaran S. Sakshi Oriol Nieto R. Duraiswami Dinesh Manocha AuLLM VLM CoGe 35 21 0 12 Oct 2023
Semantic Proximity Alignment: Towards Human Perception-consistent Audio Tagging by Aligning with Label Text Description Youbin Jeon Yanzhen Ren VLM 31 0 0 28 Sep 2023
Online Active Learning For Sound Event Detection Mark Lindsey Ankit Shah Francis Kubala R. M. Stern 26 0 0 25 Sep 2023
Joint Audio and Speech Understanding Yuan Gong Alexander H. Liu Hongyin Luo Leonid Karlinsky James R. Glass AuLLM 28 66 0 25 Sep 2023
Learning Tri-modal Embeddings for Zero-Shot Soundscape Mapping Subash Khanal S. Sastry A. Dhakal Nathan Jacobs 48 8 0 19 Sep 2023
AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes Zhaohui Li Haitao Wang Xinghua Jiang 40 1 0 14 Aug 2023
Improving Audio-Text Retrieval via Hierarchical Cross-Modal Interaction and Auxiliary Captions Yifei Xin Yuexian Zou 44 9 0 28 Jul 2023
Streaming Audio Transformers for Online Audio Tagging Heinrich Dinkel Zhiyong Yan Yongqing Wang Junbo Zhang Yujun Wang Bin Wang 34 4 0 29 May 2023
Adapting Language-Audio Models as Few-Shot Audio Learners Jinhua Liang Xubo Liu Haohe Liu Huy P Phan Emmanouil Benetos Mark D. Plumbley Wenwu Wang VLM 32 19 0 28 May 2023
Pengi: An Audio Language Model for Audio Tasks Soham Deshmukh Benjamin Elizalde Rita Singh Huaming Wang MLLM AuLLM 34 158 0 19 May 2023
ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities Peng Wang Shijie Wang Junyang Lin Shuai Bai Xiaohuan Zhou Jingren Zhou Xinggang Wang Chang Zhou VLM MLLM ObjD 42 115 0 18 May 2023
Universal Source Separation with Weakly Labelled Data Qiuqiang Kong K. Chen Haohe Liu Xingjian Du Taylor Berg-Kirkpatrick Shlomo Dubnov Mark D. Plumbley 18 17 0 11 May 2023
Transformer-based Sequence Labeling for Audio Classification based on MFCCs C. Sonali S. ChinmayiB A. Balasubramanian 32 0 0 30 Apr 2023
MMViT: Multiscale Multiview Vision Transformers Yuchen Liu Natasha Ong Kaiyan Peng Bo Xiong Qifan Wang ... Madian Khabsa Kaiyue Yang David C. Liu Donald Williamson Hanchao Yu ViT 27 4 0 28 Apr 2023
Multiscale Audio Spectrogram Transformer for Efficient Audio Classification Wenjie Zhu M. Omar 37 22 0 19 Mar 2023
CAT: Causal Audio Transformer for Audio Classification Xiaoyu Liu Hanlin Lu Jianbo Yuan Xinyu Li ViT 26 22 0 14 Mar 2023
SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing Weidong Chen Xiaofen Xing Xiangmin Xu Jianxin Pang Lan Du 30 38 0 27 Feb 2023
SemanticAC: Semantics-Assisted Framework for Audio Classification Yicheng Xiao Yue Ma Shuyan Li Hantao Zhou Ran Liao Xiu Li 13 8 0 12 Feb 2023
Revisiting Pre-training in Audio-Visual Learning Ruoxuan Feng Wenke Xia Di Hu 27 1 0 07 Feb 2023
Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation Yusong Wu K. Chen Tianyu Zhang Yuchen Hui Marianna Nezhurina Taylor Berg-Kirkpatrick Shlomo Dubnov CLIP 39 483 0 12 Nov 2022
Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation Florian Schmid Khaled Koutini Gerhard Widmer ViT 25 58 0 09 Nov 2022
SpectroMap: Peak detection algorithm for audio fingerprinting A. López-García 28 0 0 02 Nov 2022
Play It Back: Iterative Attention for Audio Recognition Alexandros Stergiou Dima Damen 37 4 0 20 Oct 2022
Learning Temporal Resolution in Spectrogram for Audio Classification Haohe Liu Xubo Liu Qiuqiang Kong Wenwu Wang Mark D. Plumbley 34 7 0 04 Oct 2022
Contrastive Audio-Visual Masked Autoencoder Yuan Gong Andrew Rouditchenko Alexander H. Liu David Harwath Leonid Karlinsky Hilde Kuehne James R. Glass 32 120 0 02 Oct 2022
An empirical study of weakly supervised audio tagging embeddings for general audio representations Heinrich Dinkel Zhiyong Yan Yongqing Wang Junbo Zhang Yujun Wang 37 1 0 30 Sep 2022
UniKW-AT: Unified Keyword Spotting and Audio Tagging Heinrich Dinkel Yongqing Wang Zhiyong Yan Junbo Zhang Yujun Wang 42 3 0 23 Sep 2022
Language-based Audio Retrieval Task in DCASE 2022 Challenge Huang Xie Samuel Lipping Tuomas Virtanen 60 18 0 20 Sep 2022