FastAST: Accelerating Audio Spectrogram Transformer via Token Merging
and Cross-Model Knowledge Distillation

FastAST: Accelerating Audio Spectrogram Transformer via Token Merging and Cross-Model Knowledge Distillation

11 June 2024

Swarup Ranjan Behera

Abhishek Dhiman

Aalekhya Satya Narayani

ArXiv (abs)PDF HTML

Papers citing "FastAST: Accelerating Audio Spectrogram Transformer via Token Merging and Cross-Model Knowledge Distillation"

15 / 15 papers shown

Title
Token Merging: Your ViT But Faster Daniel Bolya Cheng-Yang Fu Xiaoliang Dai Peizhao Zhang Christoph Feichtenhofer Judy Hoffman MoMe 117 470 0 17 Oct 2022
CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification Yuan Gong Sameer Khurana Andrew Rouditchenko James R. Glass VLM 61 29 0 13 Mar 2022
HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection Ke Chen Xingjian Du Bilei Zhu Zejun Ma Taylor Berg-Kirkpatrick Shlomo Dubnov ViT 159 273 0 02 Feb 2022
SPViT: Enabling Faster Vision Transformers via Soft Token Pruning Zhenglun Kong Peiyan Dong Xiaolong Ma Xin Meng Mengshu Sun ... Geng Yuan Bin Ren Minghai Qin Hao Tang Yanzhi Wang ViT 74 153 0 27 Dec 2021
AdaViT: Adaptive Tokens for Efficient Vision Transformer Hongxu Yin Arash Vahdat J. Álvarez Arun Mallya Jan Kautz Pavlo Molchanov ViT 118 341 0 14 Dec 2021
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition Lingchen Meng Hengduo Li Bor-Chun Chen Shiyi Lan Zuxuan Wu Yu-Gang Jiang Ser-Nam Lim ViT 89 238 0 30 Nov 2021
Efficient Training of Audio Transformers with Patchout Khaled Koutini Jan Schluter Hamid Eghbalzadeh Gerhard Widmer ViT 131 261 0 11 Oct 2021
Knowledge distillation: A good teacher is patient and consistent Lucas Beyer Xiaohua Zhai Amelie Royer L. Markeeva Rohan Anil Alexander Kolesnikov VLM 107 297 0 09 Jun 2021
AST: Audio Spectrogram Transformer Yuan Gong Yu-An Chung James R. Glass ViT 134 882 0 05 Apr 2021
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 670 41,430 0 22 Oct 2020
Rethinking CNN Models for Audio Classification Kamalesh Palanisamy Dipika Singhania Angela Yao SSL 72 145 0 22 Jul 2020
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks Mingxing Tan Quoc V. Le 3DV MedIm 153 18,179 0 28 May 2019
Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition Pete Warden 94 1,626 0 09 Apr 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 786 132,363 0 12 Jun 2017
CNN Architectures for Large-Scale Audio Classification Shawn Hershey Sourish Chaudhuri D. Ellis J. Gemmeke A. Jansen ... Rif A. Saurous Bryan Seybold M. Slaney Ron J. Weiss K. Wilson 128 2,507 0 29 Sep 2016