v1v2 (latest)

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

16 July 2021

Junnan Li

Ramprasaath R. Selvaraju

Akhilesh Deepak Gotmare

ArXiv (abs)PDF HTML Github (1658★)

Papers citing "Align before Fuse: Vision and Language Representation Learning with Momentum Distillation"

50 / 1,231 papers shown

Title
Rethinking CLIP-based Video Learners in Cross-Domain Open-Vocabulary Action Recognition Kun-Yu Lin Henghui Ding Jiaming Zhou Yu-Ming Tang Yi-Xing Peng Zhilin Zhao Chen Change Loy Wei-Shi Zheng VLM 123 18 0 03 Mar 2024
Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval Yongchao Du Min Wang Wen-gang Zhou Shuping Hui Houqiang Li 74 12 0 03 Mar 2024
Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training Haowei Liu Yaya Shi Haiyang Xu Chunfen Yuan Qinghao Ye ... Mingshi Yan Ji Zhang Fei Huang Bing Li Weiming Hu VLM 94 0 0 01 Mar 2024
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers Tsai-Shien Chen Aliaksandr Siarohin Willi Menapace Ekaterina Deyneka Hsiang-wei Chao ... Yuwei Fang Hsin-Ying Lee Jian Ren Ming-Hsuan Yang Sergey Tulyakov VGen 166 211 0 29 Feb 2024
The All-Seeing Project V2: Towards General Relation Comprehension of the Open World Weiyun Wang Yiming Ren Hao Luo Tiantong Li Chenxiang Yan ... Qingyun Li Lewei Lu Xizhou Zhu Yu Qiao Jifeng Dai MLLM 143 53 0 29 Feb 2024
Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlook Xingchen Zou Yibo Yan Xixuan Hao Yuehong Hu Haomin Wen ... Junbo Zhang Yong Li Tianrui Li Yu Zheng Yuxuan Liang HAI AI4TS 104 45 0 29 Feb 2024
How to Understand "Support"? An Implicit-enhanced Causal Inference Approach for Weakly-supervised Phrase Grounding Jiamin Luo Jianing Zhao Jingjing Wang Guodong Zhou 64 0 0 29 Feb 2024
G4G:A Generic Framework for High Fidelity Talking Face Generation with Fine-grained Intra-modal Alignment Juan Zhang Jiahao Chen Cheng Wang Zhi-Yang Yu Tangquan Qi Di Wu CVBM 78 0 0 28 Feb 2024
Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control Thong Nguyen Mariya Hendriksen Andrew Yates Maarten de Rijke 83 6 0 27 Feb 2024
Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning Maurits J. R. Bleeker Mariya Hendriksen Andrew Yates Maarten de Rijke VLM 97 2 0 27 Feb 2024
CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification Zihang Jiang Qingsong Yao Zihang Jiang Rongsheng Wang Zhiyang He Xiaodong Tao S. Kevin Zhou MedIm 96 16 0 27 Feb 2024
Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion Xuantong Liu Tianyang Hu Wei Cao Kenji Kawaguchi Yuan Yao DiffM 117 3 0 26 Feb 2024
LSTP: Language-guided Spatial-Temporal Prompt Learning for Long-form Video-Text Understanding Yuxuan Wang Yueqian Wang Pengfei Wu Jianxin Liang Dongyan Zhao Zilong Zheng VLM 93 8 0 25 Feb 2024
TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages Minsu Kim Jee-weon Jung Hyeongseop Rha Soumi Maiti Siddhant Arora Xuankai Chang Shinji Watanabe Y. Ro 102 7 0 25 Feb 2024
CLIPose: Category-Level Object Pose Estimation with Pre-trained Vision-Language Knowledge Xiao Lin Minghao Zhu Ronghao Dang Guangliang Zhou Shaolong Shu Feng Lin Chengju Liu Qi Chen CLIP 119 9 0 24 Feb 2024
Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models Chaoya Jiang Wei Ye Mengfan Dong Hongrui Jia Haiyang Xu Mingshi Yan Ji Zhang Shikun Zhang VLM MLLM 120 16 0 24 Feb 2024
Artful Path to Healing: Using Machine Learning for Visual Art Recommendation to Prevent and Reduce Post-Intensive Care B. Yilma Chan Mi Kim G. Cupchik Luis A Leiva 51 6 0 23 Feb 2024
Self-Supervised Pre-Training for Table Structure Recognition Transformer Sheng-Hsuan Peng Seongmin Lee Xiaojing Wang Rajarajeswari Balasubramaniyan Duen Horng Chau ViT LMTD 60 0 0 23 Feb 2024
Assessing News Thumbnail Representativeness: Counterfactual text can enhance the cross-modal matching ability Yejun Yoon Seunghyun Yoon Kunwoo Park 92 1 0 17 Feb 2024
PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter Junfei Xiao Zheng Xu Alan Yuille Shen Yan Boyu Wang 39 3 0 16 Feb 2024
LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition Jinyuan Li Han Li Di Sun Jiahao Wang Wenkun Zhang Zan Wang Gang Pan 101 7 0 15 Feb 2024
Align before Attend: Aligning Visual and Textual Features for Multimodal Hateful Content Detection E. Hossain Omar Sharif M. M. Hoque S. Preum 71 4 0 15 Feb 2024
ProtChatGPT: Towards Understanding Proteins with Large Language Models Chao Wang Hehe Fan Ruijie Quan Yi Yang 108 16 0 15 Feb 2024
Can Text-to-image Model Assist Multi-modal Learning for Visual Recognition with Visual Modality Missing? Tiantian Feng Daniel Yang Digbalay Bose Shrikanth Narayanan 100 6 0 14 Feb 2024
PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs Michael Dorkenwald Nimrod Barazani Cees G. M. Snoek Yuki M. Asano VLM MLLM 59 12 0 13 Feb 2024
An Empirical Study Into What Matters for Calibrating Vision-Language Models Weijie Tu Weijian Deng Dylan Campbell Stephen Gould Tom Gedeon VLM 88 8 0 12 Feb 2024
GS-CLIP: Gaussian Splatting for Contrastive Language-Image-3D Pretraining from Real-World Data Haoyuan Li Yanpeng Zhou Yihan Zeng Hang Xu Xiaodan Liang 3DGS CLIP 40 0 0 09 Feb 2024
Premier-TACO is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Loss Ruijie Zheng Yongyuan Liang Xiyao Wang Shuang Ma Hal Daumé Huazhe Xu John Langford Praveen Palanisamy Kalyan Shankar Basu Furong Huang 103 8 0 09 Feb 2024
Question Aware Vision Transformer for Multimodal Reasoning Roy Ganz Yair Kittenplon Aviad Aberdam Elad Ben Avraham Oren Nuriel Shai Mazor Ron Litman 106 23 0 08 Feb 2024
MobileVLM V2: Faster and Stronger Baseline for Vision Language Model Xiangxiang Chu Limeng Qiao Xinyu Zhang Shuang Xu Fei Wei ... Xiaofei Sun Yiming Hu Xinyang Lin Bo Zhang Chunhua Shen VLM MLLM 85 109 0 06 Feb 2024
The Landscape and Challenges of HPC Research and LLMs Le Chen Nesreen K. Ahmed Akashnil Dutta Arijit Bhattacharjee Sixing Yu ... Vy A. Vo J. P. Muñoz Ted Willke Tim Mattson Ali Jannesari AI4CE 86 21 0 03 Feb 2024
AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error Jonas Ricker Denis Lukovnikov Asja Fischer 136 40 0 31 Jan 2024
Proximity QA: Unleashing the Power of Multi-Modal Large Language Models for Spatial Proximity Analysis Jianing Li Xi Nan Ming Lu Li Du Shanghang Zhang 56 2 0 31 Jan 2024
Multi-granularity Correspondence Learning from Long-term Noisy Videos Yijie Lin Jie Zhang Zhenyu Huang Jia-Wei Liu Zujie Wen Xi Peng 143 22 0 30 Jan 2024
Beyond Image-Text Matching: Verb Understanding in Multimodal Transformers Using Guided Masking Ivana Beňová Jana Kosecka Michal Gregor Martin Tamajka Marcel Veselý Marian Simko 61 1 0 29 Jan 2024
Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization Yuhang Zang Hanlin Goh Josh Susskind Chen Huang VLM 110 13 0 29 Jan 2024
MResT: Multi-Resolution Sensing for Real-Time Control with Vision-Language Models Saumya Saxena Mohit Sharma Oliver Kroemer 85 4 0 25 Jan 2024
MM-LLMs: Recent Advances in MultiModal Large Language Models Duzhen Zhang Yahan Yu Jiahua Dong Chenxing Li Dan Su Chenhui Chu Dong Yu OffRL LRM 164 216 0 24 Jan 2024
SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval Siwei Wu Yizhi Li Kang Zhu Ge Zhang Yiming Liang ... Wenhu Chen Wenhao Huang Noura Al Moubayed Jie Fu Chenghua Lin 98 13 0 24 Jan 2024
Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss Jordan Shipard Arnold Wiliem Kien Nguyen Thanh Wei Xiang Clinton Fookes VLM CLIP 96 2 0 22 Jan 2024
Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement Cheng Li Weijian Huang Hao Yang Jiarun Liu Shanshan Wang MedIm 90 4 0 21 Jan 2024
Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images Kuofeng Gao Yang Bai Jindong Gu Shu-Tao Xia Philip Torr Zhifeng Li Wei Liu VLM 86 47 0 20 Jan 2024
CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarios Xiangshuo Qiao Xianxin Li Xiaozhe Qu Jie M. Zhang Yang Liu Yu Luo Cihang Jin Jin Ma VLM 93 0 0 19 Jan 2024
MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer Changyao Tian Xizhou Zhu Yuwen Xiong Weiyun Wang Zhe Chen ... Tong Lu Jie Zhou Hongsheng Li Yu Qiao Jifeng Dai AuLLM 145 49 0 18 Jan 2024
Improving fine-grained understanding in image-text pre-training Ioana Bica Anastasija Ilić Matthias Bauer Goker Erdogan Matko Bovsnjak ... A. Gritsenko Matthias Minderer Charles Blundell Razvan Pascanu Jovana Mitrović VLM 75 27 0 18 Jan 2024
POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images Antonín Vobecký Oriane Siméoni David Hurych Spyros Gidaris Andrei Bursuc Patrick Pérez Josef Sivic 111 35 0 17 Jan 2024
AiGen-FoodReview: A Multimodal Dataset of Machine-Generated Restaurant Reviews and Images on Social Media Alessandro Gambetti Qiwei Han DeLMO 59 4 0 16 Jan 2024
FiGCLIP: Fine-Grained CLIP Adaptation via Densely Annotated Videos S. DarshanSingh Zeeshan Khan Makarand Tapaswi VLM CLIP 67 3 0 15 Jan 2024
Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection Wei Ye Chaoya Jiang Haiyang Xu Chenhao Ye Chenliang Li Mingshi Yan Shikun Zhang Songhang Huang Fei Huang VLM 77 0 0 11 Jan 2024
MISS: A Generative Pretraining and Finetuning Approach for Med-VQA Jiawei Chen Dingkang Yang Yue Jiang Yuxuan Lei Lihua Zhang LM&MA MedIm 62 15 0 10 Jan 2024