CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation

15 June 2024

Wei Chen

Yongqi Yang

Yu Wu

Long Chen

VLM

VGen

ArXiv PDF HTML

Papers citing "CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation"

41 / 41 papers shown

Title
KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models Yongliang Wu Zonghui Li Xinting Hu Xinyu Ye Xianfang Zeng Gang Yu Wenbo Zhu Bernt Schiele Ming-Hsuan Yang Xu Yang VLM 48 0 0 22 May 2025
Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities Wei Wei Jintao Guo Shanshan Zhao Minghao Fu Lunhao Duan Guo-Hua Wang Qing-Guo Chen Zhao Xu Weihua Luo Kaifu Zhang DiffM 152 0 0 05 May 2025
LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant Wei Li Bing Hu Rui Shao Leyang Shen Liqiang Nie 62 3 0 05 Mar 2025
2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining Wenqi Zhang Hang Zhang Xin Li Jiashuo Sun Yongliang Shen Weiming Lu Deli Zhao Yueting Zhuang Lidong Bing VLM 77 2 0 01 Jan 2025
A Survey on Multimodal Benchmarks: In the Era of Large AI Models Lin Li Guikun Chen Hanrong Shi Jun Xiao Long Chen 69 10 0 21 Sep 2024
Visual Prompting in Multimodal Large Language Models: A Survey Junda Wu Zhehao Zhang Yu Xia Xintong Li Zhaoyang Xia ... Subrata Mitra Dimitris N. Metaxas Lina Yao Jingbo Shang Julian McAuley VLM LRM 71 16 0 05 Sep 2024
Towards a Unified View of Preference Learning for Large Language Models: A Survey Bofei Gao Feifan Song Yibo Miao Zefan Cai Zhiyong Yang ... Houfeng Wang Zhifang Sui Peiyi Wang Baobao Chang Baobao Chang 73 13 0 04 Sep 2024
Modeling Caption Diversity in Contrastive Vision-Language Pretraining Samuel Lavoie Polina Kirichenko Mark Ibrahim Mahmoud Assran Andrew Gordon Wilson Aaron Courville Nicolas Ballas CLIP VLM 86 23 0 30 Apr 2024
SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation Yuying Ge Sijie Zhao Jinguo Zhu Yixiao Ge Kun Yi Lin Song Chen Li Xiaohan Ding Ying Shan VLM 84 120 0 22 Apr 2024
MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer Changyao Tian Xizhou Zhu Yuwen Xiong Weiyun Wang Zhe Chen ... Tong Lu Jie Zhou Hongsheng Li Yu Qiao Jifeng Dai AuLLM 94 44 0 18 Jan 2024
A Survey of Reinforcement Learning from Human Feedback Timo Kaufmann Paul Weng Viktor Bengs Eyke Hüllermeier OffRL 13 134 0 22 Dec 2023
Generative Multimodal Models are In-Context Learners Quan-Sen Sun Yufeng Cui Xiaosong Zhang Fan Zhang Qiying Yu ... Yueze Wang Yongming Rao Jingjing Liu Tiejun Huang Xinlong Wang MLLM LRM 99 258 0 20 Dec 2023
OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation Jie An Zhengyuan Yang Linjie Li Jianfeng Wang Kevin Qinghong Lin Zicheng Liu Lijuan Wang Jiebo Luo 52 11 0 11 Oct 2023
MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens Kaizhi Zheng Xuehai He Xin Eric Wang MLLM 70 94 0 03 Oct 2023
Making LLaMA SEE and Draw with SEED Tokenizer Yuying Ge Sijie Zhao Ziyun Zeng Yixiao Ge Chen Li Xintao Wang Ying Shan 45 133 0 02 Oct 2023
DreamLLM: Synergistic Multimodal Comprehension and Creation Runpei Dong Chunrui Han Yuang Peng Zekun Qi Zheng Ge ... Hao-Ran Wei Xiangwen Kong Xiangyu Zhang Kaisheng Ma Li Yi MLLM 57 185 0 20 Sep 2023
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models Anas Awadalla Irena Gao Josh Gardner Jack Hessel Yusuf Hanafy ... Simon Kornblith Pang Wei Koh Gabriel Ilharco Mitchell Wortsman Ludwig Schmidt MLLM 73 410 0 02 Aug 2023
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis Dustin Podell Zion English Kyle Lacey A. Blattmann Tim Dockhorn Jonas Muller Joe Penna Robin Rombach 157 2,242 0 04 Jul 2023
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents Hugo Laurenccon Lucile Saulnier Léo Tronchon Stas Bekman Amanpreet Singh ... Siddharth Karamcheti Alexander M. Rush Douwe Kiela Matthieu Cord Victor Sanh 82 234 0 21 Jun 2023
Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Models Chang-rui Liu Haoning Wu Yujie Zhong Xiaoyu Zhang Yanfeng Wang Weidi Xie DiffM VLM 45 42 0 01 Jun 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 261 3,712 0 29 May 2023
Generating Images with Multimodal Language Models Jing Yu Koh Daniel Fried Ruslan Salakhutdinov MLLM 48 248 0 26 May 2023
DataComp: In search of the next generation of multimodal datasets S. Gadre Gabriel Ilharco Alex Fang J. Hayase Georgios Smyrnis ... A. Dimakis J. Jitsev Y. Carmon Vaishaal Shankar Ludwig Schmidt VLM 45 426 0 27 Apr 2023
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text Wanrong Zhu Jack Hessel Anas Awadalla S. Gadre Jesse Dodge Alex Fang Youngjae Yu Ludwig Schmidt William Yang Wang Yejin Choi VLM 49 170 0 14 Apr 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 403 13,788 0 15 Mar 2023
Language Is Not All You Need: Aligning Perception with Language Models Shaohan Huang Li Dong Wenhui Wang Y. Hao Saksham Singhal ... Johan Bjorck Vishrav Chaudhary Subhojit Som Xia Song Furu Wei VLM LRM MLLM 54 546 0 27 Feb 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 585 12,840 0 27 Feb 2023
LAION-5B: An open large-scale dataset for training next generation image-text models Christoph Schuhmann Romain Beaumont Richard Vencu Cade Gordon Ross Wightman ... Srivatsa Kundurthy Katherine Crowson Ludwig Schmidt R. Kaczmarczyk J. Jitsev VLM MLLM CLIP 111 3,355 0 16 Oct 2022
Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr ... Mikolaj Binkowski Ricardo Barreira Oriol Vinyals Andrew Zisserman Karen Simonyan MLLM VLM 266 3,458 0 29 Apr 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 562 9,009 0 28 Jan 2022
High-Resolution Image Synthesis with Latent Diffusion Models Robin Rombach A. Blattmann Dominik Lorenz Patrick Esser Bjorn Ommer 3DV 225 15,081 0 20 Dec 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 188 9,946 0 17 Jun 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 592 28,659 0 26 Feb 2021
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes Douwe Kiela Hamed Firooz Aravind Mohan Vedanuj Goswami Amanpreet Singh Pratik Ringshia Davide Testuggine 63 597 0 10 May 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 247 19,824 0 23 Oct 2019
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge Kenneth Marino Mohammad Rastegari Ali Farhadi Roozbeh Mottaghi 49 1,050 0 31 May 2019
Towards VQA Models That Can Read Amanpreet Singh Vivek Natarajan Meet Shah Yu Jiang Xinlei Chen Dhruv Batra Devi Parikh Marcus Rohrbach EgoV 47 1,174 0 18 Apr 2019
VizWiz Grand Challenge: Answering Visual Questions from Blind People Danna Gurari Qing Li Abigale Stangl Anhong Guo Chi Lin Kristen Grauman Jiebo Luo Jeffrey P. Bigham CoGe 63 831 0 22 Feb 2018
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering Yash Goyal Tejas Khot D. Summers-Stay Dhruv Batra Devi Parikh CoGe 283 3,187 0 02 Dec 2016
Improved Techniques for Training GANs Tim Salimans Ian Goodfellow Wojciech Zaremba Vicki Cheung Alec Radford Xi Chen GAN 348 8,999 0 10 Jun 2016
Microsoft COCO: Common Objects in Context Nayeon Lee Michael Maire Serge J. Belongie Lubomir Bourdev Ross B. Girshick James Hays Pietro Perona Deva Ramanan C. L. Zitnick Piotr Dollár ObjD 215 43,290 0 01 May 2014