v1v2v3v4 (latest)

Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions

13 November 2024

Papers citing "Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions"

38 / 38 papers shown

Title
3DArticCyclists: Generating Synthetic Articulated 8D Pose-Controllable Cyclist Data for Computer Vision Applications Eduardo R. Corral-Soto Yang Liu Tongtong Cao Y. Ren Liu Bingbing 117 5 0 14 Oct 2024
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models Matt Deitke Christopher Clark Sangho Lee Rohun Tripathi Yue Yang ... Noah A. Smith Hannaneh Hajishirzi Ross Girshick Ali Farhadi Aniruddha Kembhavi OSLM VLM 84 14 0 25 Sep 2024
Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models Nitzan Bitton-Guetta Aviv Slobodkin Aviya Maimon Eliya Habba Royi Rassin Yonatan Bitton Idan Szpektor Amir Globerson Yuval Elovici ReLM VLM LRM 56 6 0 28 Jul 2024
DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception Xiaotong Li Fan Zhang Haiwen Diao Yueze Wang Xinlong Wang Ling-yu Duan VLM 99 32 0 11 Jul 2024
VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs Qiucheng Wu Handong Zhao Michael Stephen Saxon T. Bui William Yang Wang Yang Zhang Shiyu Chang CoGe 76 6 0 02 Jul 2024
From Pixels to Prose: A Large Dataset of Dense Image Captions Vasu Singla Kaiyu Yue Sukriti Paul Reza Shirkavand Mayuka Jayawardhana Alireza Ganjdanesh Heng Huang A. Bhatele Gowthami Somepalli Tom Goldstein 3DV VLM 90 27 0 14 Jun 2024
LLMs can learn self-restraint through iterative self-reflection Alexandre Piché Aristides Milios Dzmitry Bahdanau Chris Pal 63 6 0 15 May 2024
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? Zorik Gekhman G. Yona Roee Aharoni Matan Eyal Amir Feder Roi Reichart Jonathan Herzig 118 134 0 09 May 2024
DOCCI: Descriptions of Connected and Contrasting Images Yasumasa Onoe Sunayana Rane Zachary Berger Yonatan Bitton Jaemin Cho ... Zarana Parekh Jordi Pont-Tuset Garrett Tanzer Su Wang Jason Baldridge 90 63 0 30 Apr 2024
Rho-1: Not All Tokens Are What You Need Zheng-Wen Lin Zhibin Gou Yeyun Gong Xiao Liu Yelong Shen ... Chen Lin Yujiu Yang Jian Jiao Nan Duan Weizhu Chen CLL 117 75 0 11 Apr 2024
Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback Hongshen Xu Zichen Zhu Situo Zhang Da Ma Shuai Fan Lu Chen Kai Yu HILM 86 45 0 27 Mar 2024
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models Yaowei Zheng Richong Zhang Junhao Zhang Yanhan Ye Zheyan Luo Zhangchi Feng Yongqiang Ma 153 547 0 20 Mar 2024
TinyLLaVA: A Framework of Small-scale Large Multimodal Models Baichuan Zhou Ying Hu Xi Weng Junlong Jia Jie Luo Xien Liu Ji Wu Lei Huang MLLM 80 112 0 22 Feb 2024
Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection Ruibo Chen Yihan Wu Lichang Chen Guodong Liu Qi He Tianyi Xiong Chenxi Liu Junfeng Guo Heng-Chiao Huang VLM 50 21 0 19 Feb 2024
Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation Xiaoying Zhang Baolin Peng Ye Tian Jingyan Zhou Lifeng Jin Linfeng Song Haitao Mi Helen Meng HILM 73 52 0 14 Feb 2024
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs Shengbang Tong Zhuang Liu Yuexiang Zhai Yi-An Ma Yann LeCun Saining Xie VLM MLLM 105 347 0 11 Jan 2024
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback M. Steyvers Yuan Yao Haoye Zhang Taiwen He Yifeng Han ... Xinyue Hu Zhiyuan Liu Hai-Tao Zheng Maosong Sun Tat-Seng Chua MLLM VLM 196 228 0 01 Dec 2023
ShareGPT4V: Improving Large Multi-Modal Models with Better Captions Lin Chen Jinsong Li Xiao-wen Dong Pan Zhang Conghui He Jiaqi Wang Feng Zhao Dahua Lin MLLM VLM 200 680 0 21 Nov 2023
Improved Baselines with Visual Instruction Tuning Haotian Liu Chunyuan Li Yuheng Li Yong Jae Lee VLM MLLM 160 2,817 0 05 Oct 2023
From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning Ming Li Yong Zhang Zhitao Li Jiuhai Chen Lichang Chen Ning Cheng Jianzong Wang Dinesh Manocha Jing Xiao 110 211 0 23 Aug 2023
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning Fuxiao Liu Kevin Qinghong Lin Linjie Li Jianfeng Wang Yaser Yacoob Lijuan Wang VLM MLLM 119 286 0 26 Jun 2023
Sources of Hallucination by Large Language Models on Inference Tasks Nick McKenna Tianyi Li Liang Cheng Mohammad Javad Hosseini Mark Johnson Mark Steedman LRM HILM 68 201 0 23 May 2023
LIMA: Less Is More for Alignment Chunting Zhou Pengfei Liu Puxin Xu Srini Iyer Jiao Sun ... Susan Zhang Gargi Ghosh M. Lewis Luke Zettlemoyer Omer Levy ALM 106 847 0 18 May 2023
DataComp: In search of the next generation of multimodal datasets S. Gadre Gabriel Ilharco Alex Fang J. Hayase Georgios Smyrnis ... A. Dimakis J. Jitsev Y. Carmon Vaishaal Shankar Ludwig Schmidt VLM 96 448 0 27 Apr 2023
Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation Sara Sarto Manuele Barraco Marcella Cornia Lorenzo Baraldi Rita Cucchiara 72 60 0 21 Mar 2023
Locating and Editing Factual Associations in GPT Kevin Meng David Bau A. Andonian Yonatan Belinkov KELM 251 1,381 0 10 Feb 2022
Proposition-Level Clustering for Multi-Document Summarization Ori Ernst Avi Caciularu Ori Shapira Ramakanth Pasunuru Joey Tianyi Zhou Jacob Goldberger Ido Dagan 60 19 0 16 Dec 2021
Finding a Balanced Degree of Automation for Summary Evaluation Shiyue Zhang Joey Tianyi Zhou 93 44 0 23 Sep 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 490 10,496 0 17 Jun 2021
$$Q^{2}$: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering$ $Q^{2}$ : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering Or Honovich Leshem Choshen Roee Aharoni Ella Neeman Idan Szpektor Omri Abend HILM 78 141 0 16 Apr 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 975 29,810 0 26 Feb 2021
Transformer Feed-Forward Layers Are Key-Value Memories Mor Geva R. Schuster Jonathan Berant Omer Levy KELM 170 847 0 29 Dec 2020
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries Alex Jinpeng Wang Kyunghyun Cho M. Lewis HILM 86 482 0 08 Apr 2020
How Much Knowledge Can You Pack Into the Parameters of a Language Model? Adam Roberts Colin Raffel Noam M. Shazeer KELM 132 893 0 10 Feb 2020
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks Nils Reimers Iryna Gurevych 1.3K 12,301 0 27 Aug 2019
BERTScore: Evaluating Text Generation with BERT Tianyi Zhang Varsha Kishore Felix Wu Kilian Q. Weinberger Yoav Artzi 352 5,868 0 21 Apr 2019
Object Hallucination in Image Captioning Anna Rohrbach Lisa Anne Hendricks Kaylee Burns Trevor Darrell Kate Saenko 194 443 0 06 Sep 2018
Microsoft COCO Captions: Data Collection and Evaluation Server Xinlei Chen Hao Fang Nayeon Lee Ramakrishna Vedantam Saurabh Gupta Piotr Dollar C. L. Zitnick 224 2,493 0 01 Apr 2015