Visual Instruction Tuning

17 April 2023

Papers citing "Visual Instruction Tuning"

50 / 3,311 papers shown

Title
Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach Shijian Deng Wentian Zhao Yu-Jhe Li Kun Wan Daniel Miranda Ajinkya Kale Yapeng Tian LRM 93 6 0 26 Nov 2024
COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection Jinqi Xiao S. Sang Tiancheng Zhi Jing Liu Qing Yan Linjie Luo Bo Yuan Bo Yuan VLM 117 1 0 26 Nov 2024
SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation Claudia Cuttano Gabriele Trivigno Gabriele Rosi Carlo Masone Giuseppe Averta VOS 116 2 0 26 Nov 2024
DiagramQG: Concept-Focused Diagram Question Generation via Hierarchical Knowledge Integration Xinyu Zhang L. Zhang Yanrui Wu Muye Huang Wenjun Wu Bo Li Shaowei Wang Jun Liu Jun Liu 71 0 0 26 Nov 2024
Factorized Visual Tokenization and Generation Zechen Bai Jianxiong Gao Ziteng Gao Pichao Wang Zheng Zhang Tong He Mike Zheng Shou 95 3 0 25 Nov 2024
Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis Boming Miao Cuiping Li Xiaobei Wang Andi Zhang Rui Sun Zizhe Wang Yao Zhu DiffM 86 0 0 25 Nov 2024
Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation Jungeun Kim Hyeongwoo Jeon Jongseong Bae Ha Young Kim SLR 93 0 0 25 Nov 2024
MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Image Shangwen Wang Chengxiang He Huijun Liu Shan Zhao Chengyu Wang ... Xiaopeng Li Qian Wan Jun Ma Jie Yu Xiaoguang Mao VLM 96 1 0 25 Nov 2024
Language Driven Occupancy Prediction Zhu Yu Bowen Pang Lizhe Liu Runmin Zhang Qihao Peng Maochun Luo Sheng Yang Mingxia Chen Si-Yuan Cao Hui-Liang Shen 108 2 0 25 Nov 2024
ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration Haozhan Shen Kangjia Zhao Tiancheng Zhao Ruochen Xu Zilun Zhang Mingwei Zhu Yuxiang Cai 97 4 0 25 Nov 2024
VideoOrion: Tokenizing Object Dynamics in Videos Yicheng Feng Yijiang Li Wanpeng Zhang Sipeng Zheng Zongqing Lu Sipeng Zheng Zongqing Lu 109 1 0 25 Nov 2024
RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics Chan Hee Song Valts Blukis Jonathan Tremblay Stephen Tyree Yu-Chuan Su Stan Birchfield 103 9 0 25 Nov 2024
SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAE Yongwei Chen Yushi Lan Shangchen Zhou Tengfei Wang Xingang Pan 112 5 0 25 Nov 2024
All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages Ashmal Vayani Dinura Dissanayake Hasindri Watawana Noor Ahsan Nevasini Sasikumar ... Monojit Choudhury Ivan Laptev Mubarak Shah Salman Khan Fahad A Khan 128 9 0 25 Nov 2024
GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis Bo Liu K. Zou Liming Zhan Zexin Lu Xiaoyu Dong Yidi Chen Chengqiang Xie Jiannong Cao Xiao-Ming Wu Huazhu Fu 139 0 0 25 Nov 2024
Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks Peng Xie Yequan Bie Jianda Mao Yangqiu Song Yang Wang Hao Chen Kani Chen AAML 86 1 0 24 Nov 2024
ROOT: VLM based System for Indoor Scene Understanding and Beyond Yonghui Wang Shi-Yong Chen Zhenxing Zhou Siyi Li Haoran Li Wengang Zhou Haoyang Li VLM 78 3 0 24 Nov 2024
Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuning Ji Hyeok Jung Eun Tae Kim S. Kim Joo Ho Lee Bumsoo Kim Buru Chang VLM 335 0 0 24 Nov 2024
AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea Qifan Yu Wei Chow Zhongqi Yue Kaihang Pan Yang Wu Xiaoyang Wan Juncheng Billy Li Siliang Tang Hao Zhang Yueting Zhuang DiffM 117 18 0 24 Nov 2024
freePruner: A Training-free Approach for Large Multimodal Model Acceleration Bingxin Xu Yuzhang Shang Yunhao Ge Qian Lou Yan Yan 104 3 0 23 Nov 2024
$$\textit{Revelio}$: Interpreting and leveraging semantic information in diffusion models$ $\textit{Revelio}$ : Interpreting and leveraging semantic information in diffusion models Dahye Kim Xavier Thomas Deepti Ghadiyaram 96 4 0 23 Nov 2024
FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity Hang Hua Qing Liu Lingzhi Zhang Jing Shi Zhifei Zhang Yilin Wang Jianming Zhang Jiebo Luo CoGe VLM 103 6 0 23 Nov 2024
Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts Qizhou Chen Chengyu Wang Dakan Wang Taolin Zhang Wangyue Li Xiaofeng He KELM 96 1 0 23 Nov 2024
Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens Zhangqi Jiang Junkai Chen Beier Zhu Tingjin Luo Yankun Shen Xu Yang 108 4 0 23 Nov 2024
Large Language Model with Region-guided Referring and Grounding for CT Report Generation Zhaoyu Chen Yequan Bie Haibo Jin Hao Chen 288 0 0 23 Nov 2024
ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models Junzhe Chen Tianshu Zhang Shijie Huang Yuwei Niu Linfeng Zhang Lijie Wen Xuming Hu MLLM VLM 323 2 0 22 Nov 2024
Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification S P Sharan Minkyu Choi Sahil Shah Harsh Goel Mohammad Omama Sandeep Chinchali EGVM 110 3 0 22 Nov 2024
FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression Yuke Zhu Chi Xie Shuang Liang Bo Zheng Sheng Guo 90 9 0 21 Nov 2024
Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts Honglin Li Yuting Gao Chenglu Zhu Jingdong Chen M. Yang Lin Yang MLLM 124 0 0 21 Nov 2024
IterIS: Iterative Inference-Solving Alignment for LoRA Merging Hongxu Chen Runshi Li Bowei Zhu Zhen Wang Long Chen MoMe 100 0 0 21 Nov 2024
FuseGPT: Learnable Layers Fusion of Generative Pre-trained Transformers Zehua Pei Hui-Ling Zhen Xianzhi Yu Sinno Jialin Pan Mingxuan Yuan Bei Yu AI4CE 94 1 0 21 Nov 2024
Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios Shantanu Jaiswal Debaditya Roy Basura Fernando Cheston Tan ReLM LRM 86 2 0 20 Nov 2024
DriveMLLM: A Benchmark for Spatial Understanding with Multimodal Large Language Models in Autonomous Driving Xianda Guo Ruijun Zhang Yiqun Duan Yuhang He Chenming Zhang Shuai Liu Long Chen LRM 107 12 0 20 Nov 2024
LMM-driven Semantic Image-Text Coding for Ultra Low-bitrate Learned Image Compression Shimon Murai Heming Sun J. Katto VLM 85 1 0 20 Nov 2024
ORID: Organ-Regional Information Driven Framework for Radiology Report Generation Tiancheng Gu Kaicheng Yang Xiang An Ziyong Feng Dongnan Liu Weidong Cai 94 1 0 20 Nov 2024
Teaching VLMs to Localize Specific Objects from In-context Examples Sivan Doveh Nimrod Shabtay Wei Lin Eli Schwartz Hilde Kuehne ... Leonid Karlinsky James Glass Assaf Arbelle S. Ullman Muhammad Jehanzeb Mirza VLM 111 1 0 20 Nov 2024
Visual-Oriented Fine-Grained Knowledge Editing for MultiModal Large Language Models Zhen Zeng Leijiang Gu Xun Yang Zhangling Duan Zenglin Shi Meng Wang KELM 87 2 0 19 Nov 2024
Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model Yiming Shi Xun Zhu Ying Hu Chenyi Guo Miao Li Ji Wu 91 1 0 19 Nov 2024
Generative Timelines for Instructed Visual Assembly Alejandro Pardo Jui-hsien Wang Guohao Li Josef Sivic Bryan C. Russell Fabian Caba Heilbron VGen 74 0 0 19 Nov 2024
VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge Vishwesh Nath Wenqi Li Dong Yang Andriy Myronenko Mingxin Zheng ... Holger Roth Daguang Xu Baris Turkbey Holger Roth Daguang Xu VLM 121 4 0 19 Nov 2024
TrojanRobot: Physical-World Backdoor Attacks Against VLM-based Robotic Manipulation Xiaobei Wang Hewen Pan Hangtao Zhang Minghui Li Shengshan Hu ... Peijin Guo Yichen Wang Wei Wan Aishan Liu L. Zhang AAML 107 7 0 18 Nov 2024
VLN-Game: Vision-Language Equilibrium Search for Zero-Shot Semantic Navigation Bangguo Yu Yuzhen Liu Lei Han Hamidreza Kasaei Tingguang Li M. Cao LM&Ro 89 3 0 18 Nov 2024
PSA-VLM: Enhancing Vision-Language Model Safety through Progressive Concept-Bottleneck-Driven Alignment Zhendong Liu Yuanbi Nie Yingshui Tan Xiangyu Yue Qiushi Cui Chongjun Wang Xiaoyong Zhu Jian Xu Bo Zheng 83 0 0 18 Nov 2024
SignEye: Traffic Sign Interpretation from Vehicle First-Person View Chuang Yang Xu Han T. Han Yuejiao Su Junyu Gao Hongyuan Zhang Yi Wang Lap-Pui Chau 91 2 0 18 Nov 2024
AtomThink: A Slow Thinking Framework for Multimodal Mathematical Reasoning Kun Xiang Zhili Liu Zihao Jiang Yunshuang Nie Runhui Huang ... Yihan Zeng Jiawei Han Lanqing Hong Hang Xu Xiaodan Liang LRM 130 10 0 18 Nov 2024
GLDesigner: Leveraging Multi-Modal LLMs as Designer for Enhanced Aesthetic Text Glyph Layouts Junwen He Yifan Wang Lijun Wang Huchuan Lu Jun-Yan He Chong Li Hanyuan Chen Jin-Peng Lan Bin Luo Yifeng Geng 79 1 0 18 Nov 2024
MC-LLaVA: Multi-Concept Personalized Vision-Language Model Ruichuan An Sihan Yang Ming Lu Kai Zeng Yulin Luo ... Hao Liang Qi She Shanghang Zhang Wentao Zhang Wentao Zhang 107 7 0 18 Nov 2024
Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics Taowen Wang Dongfang Liu James Liang Wenhao Yang Qifan Wang Cheng Han Jiebo Luo Ruixiang Tang Ruixiang Tang AAML 100 4 0 18 Nov 2024
SoK: Unifying Cybersecurity and Cybersafety of Multimodal Foundation Models with an Information Theory Approach Ruoxi Sun Jiamin Chang Hammond Pearce Chaowei Xiao B. Li Qi Wu Surya Nepal Minhui Xue 57 0 0 17 Nov 2024
TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models Tingyu Qu Mingxiao Li Tinne Tuytelaars Marie-Francine Moens VLM 52 2 0 17 Nov 2024