Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language

14 December 2022

Papers citing "Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language"

50 / 59 papers shown

Title
MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representations Benedikt Alkin Lukas Miklautz Sepp Hochreiter Johannes Brandstetter VLM 138 8 0 24 Feb 2025
CR-CTC: Consistency regularization on CTC for improved speech recognition Zengwei Yao Wei Kang Xiaoyu Yang Fangjun Kuang Liyong Guo Han Zhu Zengrui Jin Zhaoqing Li Long Lin Daniel Povey 74 2 0 17 Feb 2025
Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation Sungnyun Kim Sungwoo Cho Sangmin Bae Kangwook Jang Se-Young Yun SSL 97 1 0 23 Jan 2025
MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentation Taha Koleilat Hojat Asgariandehkordi H. Rivaz Yiming Xiao MedIm VLM 65 8 0 28 Sep 2024
Towards Sustainable Self-supervised Learning Shanghua Gao Pan Zhou Mingg-Ming Cheng Shuicheng Yan CLL 87 7 0 20 Oct 2022
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers Zhiliang Peng Li Dong Hangbo Bao QiXiang Ye Furu Wei 40 314 0 12 Aug 2022
OmniMAE: Single Model Masked Pretraining on Images and Videos Rohit Girdhar Alaaeldin El-Nouby Mannat Singh Kalyan Vasudev Alwala Armand Joulin Ishan Misra ViT 64 99 0 16 Jun 2022
Masked Siamese ConvNets L. Jing Jiachen Zhu Yann LeCun SSL 74 35 0 15 Jun 2022
Emergent Abilities of Large Language Models Jason W. Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph ... Tatsunori Hashimoto Oriol Vinyals Percy Liang J. Dean W. Fedus ELM ReLM LRM 170 2,428 0 15 Jun 2022
Extreme Masking for Learning Instance and Distributed Visual Representations Zhirong Wu Zihang Lai Xiao Sun Stephen Lin 66 22 0 09 Jun 2022
Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERT Bowen Shi Abdel-rahman Mohamed Wei-Ning Hsu SSL 44 18 0 15 May 2022
Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr ... Mikolaj Binkowski Ricardo Barreira Oriol Vinyals Andrew Zisserman Karen Simonyan MLLM VLM 268 3,458 0 29 Apr 2022
On-demand compute reduction with stochastic wav2vec 2.0 Apoorv Vyas Wei-Ning Hsu Michael Auli Alexei Baevski 34 13 0 25 Apr 2022
Masked Siamese Networks for Label-Efficient Learning Mahmoud Assran Mathilde Caron Ishan Misra Piotr Bojanowski Florian Bordes Pascal Vincent Armand Joulin Michael G. Rabbat Nicolas Ballas SSL 56 314 0 14 Apr 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 114 1,894 0 29 Mar 2022
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language Alexei Baevski Wei-Ning Hsu Qiantong Xu Arun Babu Jiatao Gu Michael Auli SSL VLM ViT 70 845 0 07 Feb 2022
Masked Feature Prediction for Self-Supervised Visual Pre-Training Chen Wei Haoqi Fan Saining Xie Chaoxia Wu Alan Yuille Christoph Feichtenhofer ViT 120 661 0 16 Dec 2021
FLAVA: A Foundational Language And Vision Alignment Model Amanpreet Singh Ronghang Hu Vedanuj Goswami Guillaume Couairon Wojciech Galuba Marcus Rohrbach Douwe Kiela CLIP VLM 62 695 0 08 Dec 2021
SimMIM: A Simple Framework for Masked Image Modeling Zhenda Xie Zheng Zhang Yue Cao Yutong Lin Jianmin Bao Zhuliang Yao Qi Dai Han Hu 154 1,331 0 18 Nov 2021
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale Arun Babu Changhan Wang Andros Tjandra Kushal Lakhotia Qiantong Xu ... Yatharth Saraf J. Pino Alexei Baevski Alexis Conneau Michael Auli SSL 61 678 0 17 Nov 2021
iBOT: Image BERT Pre-Training with Online Tokenizer Jinghao Zhou Chen Wei Huiyu Wang Wei Shen Cihang Xie Alan Yuille Tao Kong 59 722 0 15 Nov 2021
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 361 7,600 0 11 Nov 2021
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts Hangbo Bao Wenhui Wang Li Dong Qiang Liu Owais Khan Mohammed Kriti Aggarwal Subhojit Som Furu Wei VLM MLLM MoE 54 543 0 03 Nov 2021
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing Sanyuan Chen Chengyi Wang Zhengyang Chen Yu-Huan Wu Shujie Liu ... Yao Qian Jian Wu Micheal Zeng Xiangzhan Yu Furu Wei SSL 171 1,794 0 26 Oct 2021
Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition Felix Wu Kwangyoun Kim Jing Pan Kyu Jeong Han Kilian Q. Weinberger Yoav Artzi 29 71 0 14 Sep 2021
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 278 728 0 27 Aug 2021
Perceiver IO: A General Architecture for Structured Inputs & Outputs Andrew Jaegle Sebastian Borgeaud Jean-Baptiste Alayrac Carl Doersch Catalin Ionescu ... Olivier J. Hénaff M. Botvinick Andrew Zisserman Oriol Vinyals João Carreira MLLM VLM GNN 40 571 0 30 Jul 2021
Efficient Self-supervised Vision Transformers for Representation Learning Chunyuan Li Jianwei Yang Pengchuan Zhang Mei Gao Bin Xiao Xiyang Dai Lu Yuan Jianfeng Gao ViT 73 213 0 17 Jun 2021
BEiT: BERT Pre-Training of Image Transformers Hangbo Bao Li Dong Songhao Piao Furu Wei ViT 156 2,785 0 15 Jun 2021
Unsupervised Speech Recognition Alexei Baevski Wei-Ning Hsu Alexis Conneau Michael Auli SSL 93 273 0 24 May 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 542 5,920 0 29 Apr 2021
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text Hassan Akbari Liangzhe Yuan Rui Qian Wei-Hong Chuang Shih-Fu Chang Huayu Chen Boqing Gong ViT 260 581 0 22 Apr 2021
An Empirical Study of Training Self-Supervised Vision Transformers Xinlei Chen Saining Xie Kaiming He ViT 114 1,837 0 05 Apr 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 122 999 0 04 Mar 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 605 28,659 0 26 Feb 2021
CLEAR: Contrastive Learning for Sentence Representation Zhuofeng Wu Sinong Wang Jiatao Gu Madian Khabsa Fei Sun Hao Ma SSL 45 319 0 31 Dec 2020
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech Andy T. Liu Shang-Wen Li Hung-yi Lee SSL 96 356 0 12 Jul 2020
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations Alexei Baevski Henry Zhou Abdel-rahman Mohamed Michael Auli SSL 140 5,677 0 20 Jun 2020
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments Mathilde Caron Ishan Misra Julien Mairal Priya Goyal Piotr Bojanowski Armand Joulin OCL SSL 119 4,032 0 17 Jun 2020
Bootstrap your own latent: A new approach to self-supervised Learning Jean-Bastien Grill Florian Strub Florent Altché Corentin Tallec Pierre Harvey Richemond ... M. G. Azar Bilal Piot Koray Kavukcuoglu Rémi Munos Michal Valko SSL 251 6,718 0 13 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 445 41,106 0 28 May 2020
Libri-Light: A Benchmark for ASR with Limited or No Supervision Jacob Kahn M. Rivière Weiyi Zheng Evgeny Kharitonov Qiantong Xu ... Tatiana Likhomanenko Gabriel Synnaeve Armand Joulin Abdel-rahman Mohamed Emmanuel Dupoux AuLLM 32 660 0 17 Dec 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 249 19,824 0 23 Oct 2019
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations Alexei Baevski Steffen Schneider Michael Auli SSL 73 662 0 12 Oct 2019
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut SSL AIMat 244 6,420 0 26 Sep 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 382 24,160 0 26 Jul 2019
Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks Ryan Eloff A. Nortje Benjamin van Niekerk Avashna Govender Leanne Nortje Arnu Pretorius Elan Van Biljon Ewald van der Westhuizen Lisa van Staden Herman Kamper DRL 43 57 0 16 Apr 2019
fairseq: A Fast, Extensible Toolkit for Sequence Modeling Myle Ott Sergey Edunov Alexei Baevski Angela Fan Sam Gross Nathan Ng David Grangier Michael Auli VLM FaML 71 3,141 0 01 Apr 2019
Cloze-driven Pretraining of Self-attention Networks Alexei Baevski Sergey Edunov Yinhan Liu Luke Zettlemoyer Michael Auli 30 198 0 19 Mar 2019
Representation Learning with Contrastive Predictive Coding Aaron van den Oord Yazhe Li Oriol Vinyals DRL SSL 215 10,152 0 10 Jul 2018