v1v2 (latest)

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

16 July 2021

Junnan Li

Ramprasaath R. Selvaraju

Akhilesh Deepak Gotmare

ArXiv (abs)PDF HTML Github (1658★)

Papers citing "Align before Fuse: Vision and Language Representation Learning with Momentum Distillation"

50 / 1,231 papers shown

Title
mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus Matthieu Futeral A. Zebaze Pedro Ortiz Suarez Julien Abadji Rémi Lacroix Cordelia Schmid Rachel Bawden Benoît Sagot 167 3 0 13 Jun 2024
ConceptHash: Interpretable Fine-Grained Hashing via Concept Discovery Kam Woh Ng Xiatian Zhu Yi-Zhe Song Tao Xiang 84 2 0 12 Jun 2024
ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs Irene Huang Wei Lin M. Jehanzeb Mirza Jacob A. Hansen Sivan Doveh ... Trevor Darrel Chuang Gan Aude Oliva Rogerio Feris Leonid Karlinsky CoGe LRM 88 9 0 12 Jun 2024
Labeling Comic Mischief Content in Online Videos with a Multimodal Hierarchical-Cross-Attention Model Elaheh Baharlouei Mahsa Shafaei Yigeng Zhang Hugo Jair Escalante Thamar Solorio 79 0 0 12 Jun 2024
Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions Renjie Pi Jianshu Zhang Jipeng Zhang Boyao Wang Zhekai Chen Tong Zhang 3DV 87 24 0 11 Jun 2024
Benchmarking Vision-Language Contrastive Methods for Medical Representation Learning Shuvendu Roy Yasaman Parhizkar Franklin Ogidi Vahid Reza Khazaie Michael Colacci Ali Etemad Elham Dolatabadi Arash Afkanpour VLM 128 1 0 11 Jun 2024
Learning Domain-Invariant Features for Out-of-Context News Detection Yimeng Gu Mengqi Zhang Ignacio Castro Shu Wu Gareth Tyson 98 2 0 11 Jun 2024
Advancing Grounded Multimodal Named Entity Recognition via LLM-Based Reformulation and Box-Based Segmentation Jinyuan Li Ziyan Li Han Li Jianfei Yu Rui Xia Di Sun Gang Pan 67 2 0 11 Jun 2024
BrainChat: Decoding Semantic Information from fMRI using Vision-language Pretrained Models Wanaiu Huang 61 2 0 10 Jun 2024
Gentle-CLIP: Exploring Aligned Semantic In Low-Quality Multimodal Data With Soft Alignment Zijia Song Z. Zang Yelin Wang Guozheng Yang Jiangbin Zheng Kaicheng Yu Wanyu Chen Stan Z. Li 75 1 0 09 Jun 2024
Generalist Multimodal AI: A Review of Architectures, Challenges and Opportunities Sai Munikoti Ian Stewart Sameera Horawalavithana Henry Kvinge Tegan H. Emerson Sandra E Thompson Karl Pazdernik 102 2 0 08 Jun 2024
ChatSR: Multimodal Large Language Models for Scientific Formula Discovery Yanjie Li Weijun Li Lina Yu Min Wu Jingyi Liu Wenqiang Li Shu Wei Yusong Deng OffRL 96 3 0 08 Jun 2024
One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models Hao Fang Jiawei Kong Wenbo Yu Bin Chen Jiawei Li Hao Wu Ke Xu Ke Xu AAML VLM 133 14 0 08 Jun 2024
Low-Rank Similarity Mining for Multimodal Dataset Distillation Yue Xu Zhilin Lin Yusong Qiu Cewu Lu Yong-Lu Li DD 102 6 0 06 Jun 2024
Attribute-Aware Implicit Modality Alignment for Text Attribute Person Search Xin Wang Fangfang Liu Zheng Li Caili Guo 100 1 0 06 Jun 2024
Wings: Learning Multimodal LLMs without Text-only Forgetting Yi-Kai Zhang Shiyin Lu Yang Li Yanqing Ma Qing-Guo Chen Zhao Xu Weihua Luo Kaifu Zhang De-Chuan Zhan Han-Jia Ye VLM 126 10 0 05 Jun 2024
Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models Jinhao Li Haopeng Li S. Erfani Lei Feng James Bailey Feng Liu VLM 106 6 0 05 Jun 2024
Diffusion-Refined VQA Annotations for Semi-Supervised Gaze Following Qiaomu Miao Alexandros Graikos Jingwei Zhang Sounak Mondal Minh Hoai Dimitris Samaras 149 0 0 04 Jun 2024
Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking Zefeng Zhang Shuaiyi Nie Chuang Zhang Yunzhi Liang Wenyuan Zhang Siqi Wang Tingwen Liu OT 101 3 0 04 Jun 2024
Progressive Confident Masking Attention Network for Audio-Visual Segmentation Yuxuan Wang Feng Dong Jinchao Zhu Shuyue Zhu VOS 161 0 0 04 Jun 2024
OLIVE: Object Level In-Context Visual Embeddings Timothy Ossowski Junjie Hu OCL VLM 101 0 0 02 Jun 2024
Diversifying Query: Region-Guided Transformer for Temporal Sentence Grounding Xiaolong Sun Liushuai Shi Le Wang Sanpin Zhou Kun Xia Yabing Wang Gang Hua 88 2 0 31 May 2024
Don't Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models A. Bavaresco A. Testoni Raquel Fernández 73 2 0 31 May 2024
Knowledge-grounded Adaptation Strategy for Vision-language Models: Building Unique Case-set for Screening Mammograms for Residents Training Aisha Urooj Khan John W. Garrett Tyler Bradshaw Lonie R. Salkowski Jiwoong Jeong Amara Tariq Imon Banerjee VLM 78 2 0 30 May 2024
Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training Jinxia Yang Fuchun Sun Wayne Xin Zhao Ji-Rong Wen 96 4 0 30 May 2024
Evaluating Vision-Language Models on Bistable Images Artemis Panagopoulou Coby Melkin Chris Callison-Burch 65 0 0 29 May 2024
ContextBLIP: Doubly Contextual Alignment for Contrastive Image Retrieval from Linguistically Complex Descriptions Honglin Lin Siyu Li Gu Nan Chaoyue Tang Xueting Wang ... Yankai Rong Zhili Zhou Yutong Gao Qimei Cui Xiaofeng Tao 50 0 0 29 May 2024
Topological Perspectives on Optimal Multimodal Embedding Spaces Abdul Aziz Abdul Rahim BDL 78 0 0 29 May 2024
MetaToken: Detecting Hallucination in Image Descriptions by Meta Classification Laura Fieback Jakob Spiegelberg Hanno Gottschalk MLLM 232 5 0 29 May 2024
Dataset Growth Ziheng Qin Zhaopan Xu Yukun Zhou Zangwei Zheng Zebang Cheng ... Xiaojiang Peng Radu Timofte Hongxun Yao Kai Wang Yang You DD 43 2 0 28 May 2024
Multi-level Interaction Modeling for Protein Mutational Effect Prediction Yuanle Mo Xin Hong Bowen Gao Yinjun Jia Yanyan Lan AI4CE 58 3 0 28 May 2024
Instruct-ReID++: Towards Universal Purpose Instruction-Guided Person Re-identification Weizhen He Yiheng Deng Yunfeng Yan Feng Zhu Yizhou Wang Lei Bai Qingsong Xie Donglian Qi Wanli Ouyang Shixiang Tang 162 3 0 28 May 2024
Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View Jin Wang Shichao Dong Yapeng Zhu Kelu Yao Weidong Zhao Chao Li Ping Luo CoGe LRM 83 2 0 27 May 2024
Do Vision-Language Transformers Exhibit Visual Commonsense? An Empirical Study of VCR Zhenyang Li Yangyang Guo Ke-Jyun Wang Xiaolin Chen Liqiang Nie Mohan S. Kankanhalli LRM 52 8 0 27 May 2024
Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Tasks Yunqi Zhang Songda Li Chunyuan Deng Luyi Wang Hui Zhao 117 0 0 27 May 2024
Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs Mustafa Shukor Matthieu Cord 141 5 0 26 May 2024
CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection Lin Zhu Yifeng Yang Qinying Gu Xinbing Wang Cheng Zhou Nanyang Ye VLM 112 2 0 26 May 2024
Accelerating Transformers with Spectrum-Preserving Token Merging Hoai-Chau Tran D. M. Nguyen Duy M. Nguyen Trung Thanh Nguyen Ngan Le Pengtao Xie Daniel Sonntag James Y. Zou Binh T. Nguyen Mathias Niepert 106 13 0 25 May 2024
From Orthogonality to Dependency: Learning Disentangled Representation for Multi-Modal Time-Series Sensing Signals Ruichu Cai Zhifan Jiang Zijian Li Weilin Chen Xuexin Chen Zhifeng Hao Yifan Shen Guan-Hong Chen Kun Zhang 125 1 0 25 May 2024
LAM3D: Large Image-Point-Cloud Alignment Model for 3D Reconstruction from Single Image Ruikai Cui Xibin Song Weixuan Sun Senbo Wang Weizhe Liu ... Taizhang Shang Yang Li Nick Barnes Hongdong Li Pan Ji 3DV 75 5 0 24 May 2024
ProtFAD: Introducing function-aware domains as implicit modality towards protein function perception Mingqing Wang Zhiwei Nie Yonghong He Zhixiang Ren 57 0 0 24 May 2024
Distilling Vision-Language Pretraining for Efficient Cross-Modal Retrieval Young Kyun Jang Donghyun Kim Ser-nam Lim VLM 57 0 0 23 May 2024
Towards Transferable Attacks Against Vision-LLMs in Autonomous Driving with Typography N. Chung Sensen Gao Tuan-Anh Vu Jie M. Zhang Aishan Liu Yun Lin Jin Song Dong Qi Guo AAML 103 11 0 23 May 2024
A Survey on Vision-Language-Action Models for Embodied AI Yueen Ma Zixing Song Yuzheng Zhuang Jianye Hao Irwin King LM&Ro 330 54 0 23 May 2024
Data Augmentation for Text-based Person Retrieval Using Large Language Models Zheng Li Lijia Si Caili Guo Yang Yang Qiushi Cao 76 4 0 20 May 2024
Transcriptomics-guided Slide Representation Learning in Computational Pathology Guillaume Jaume Lukas Oldenburg Anurag J. Vaidya Richard J. Chen Drew F. K. Williamson Thomas Peeters Andrew H. Song Faisal Mahmood 112 30 0 19 May 2024
Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models Canshi Wei VLM 62 0 0 18 May 2024
Automated Radiology Report Generation: A Review of Recent Advances Phillip Sloan Philip Clatworthy Edwin Simpson Majid Mirmehdi 79 21 0 17 May 2024
FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models Adrian Bulat Yassine Ouali Georgios Tzimiropoulos VLM 104 5 0 16 May 2024
Adversarial Robustness for Visual Grounding of Multimodal Large Language Models Kuofeng Gao Yang Bai Jiawang Bai Yong Yang Shu-Tao Xia AAML 95 19 0 16 May 2024