DWIM: Towards Tool-aware Visual Reasoning via Discrepancy-aware Workflow Generation & Instruct-Masking Tuning

25 March 2025

Papers citing "DWIM: Towards Tool-aware Visual Reasoning via Discrepancy-aware Workflow Generation & Instruct-Masking Tuning"

26 / 26 papers shown

Title
NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples Baiqi Li Zhiqiu Lin Wenxuan Peng Jean de Dieu Nyandwi Daniel Jiang Zixian Ma Simran Khanuja Ranjay Krishna Graham Neubig Deva Ramanan AAML CoGe VLM 160 31 0 18 Oct 2024
Agent Planning with World Knowledge Model Shuofei Qiao Runnan Fang Ningyu Zhang Yuqi Zhu Xiang Chen Shumin Deng Yong Jiang Pengjun Xie Fei Huang Huajun Chen LLMAG LM&Ro 140 24 0 23 May 2024
FlowMind: Automatic Workflow Generation with LLMs Zhen Zeng William Watson Nicole Cho Saba Rahimi Shayleen Reynolds T. Balch Manuela Veloso 63 31 0 17 Mar 2024
KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents Yuqi Zhu Shuofei Qiao Yixin Ou Shumin Deng N. Zhang Shiwei Lyu Yue Shen Lei Liang Jinjie Gu Ningyu Zhang LLMAG LM&Ro 142 32 0 05 Mar 2024
A Survey on Multimodal Large Language Models for Autonomous Driving Can Cui Yunsheng Ma Xu Cao Wenqian Ye Yang Zhou ... Xinrui Yan Shuqi Mei Jianguo Cao Ziran Wang Chao Zheng 117 288 0 21 Nov 2023
Exploring Question Decomposition for Zero-Shot VQA Zaid Khan B. Vijaykumar S. Schulter Manmohan Chandraker Yun Fu ReLM 49 12 0 25 Oct 2023
ExpeL: LLM Agents Are Experiential Learners Andrew Zhao Daniel Huang Quentin Xu Matthieu Lin Yang Liu Gao Huang LLMAG 101 223 0 20 Aug 2023
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models Haoxuan You Rui Sun Zhecan Wang Long Chen Gengyu Wang Hammad A. Ayyubi Kai-Wei Chang Shih-Fu Chang VLM MLLM LRM 120 44 0 24 May 2023
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning Wenliang Dai Junnan Li Dongxu Li A. M. H. Tiong Junqi Zhao Weisheng Wang Boyang Albert Li Pascale Fung Steven C. H. Hoi MLLM VLM 127 2,095 0 11 May 2023
Visual Instruction Tuning Haotian Liu Chunyuan Li Qingyang Wu Yong Jae Lee SyDa VLM MLLM 560 4,861 0 17 Apr 2023
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face Yongliang Shen Kaitao Song Xu Tan Dongsheng Li Weiming Lu Yueting Zhuang MLLM 120 908 0 30 Mar 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,631 0 15 Mar 2023
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection Shilong Liu Zhaoyang Zeng Tianhe Ren Feng Li Hao Zhang ... Chun-yue Li Jianwei Yang Hang Su Jun Zhu Lei Zhang ObjD 187 2,015 0 09 Mar 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 429 4,641 0 30 Jan 2023
HiTSKT: A Hierarchical Transformer Model for Session-Aware Knowledge Tracing Fucai Ke Weiqing Wang Weicong Tan Lan Du Yuan Jin Yujin Huang Hongzhi Yin AI4Ed 43 23 0 23 Dec 2022
ReAct: Synergizing Reasoning and Acting in Language Models Shunyu Yao Jeffrey Zhao Dian Yu Nan Du Izhak Shafran Karthik Narasimhan Yuan Cao LLMAG ReLM LRM 429 2,946 0 06 Oct 2022
A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge Dustin Schwenk Apoorv Khandelwal Christopher Clark Kenneth Marino Roozbeh Mottaghi 67 551 0 03 Jun 2022
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality Tristan Thrush Ryan Jiang Max Bartolo Amanpreet Singh Adina Williams Douwe Kiela Candace Ross CoGe 106 427 0 07 Apr 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 515 6,279 0 05 Apr 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 880 13,148 0 04 Mar 2022
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 477 10,496 0 17 Jun 2021
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 835 42,332 0 28 May 2020
TallyQA: Answering Complex Counting Questions Manoj Acharya Kushal Kafle Christopher Kanan 57 125 0 29 Oct 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,114 0 11 Oct 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 728 132,199 0 12 Jun 2017
Sequence-to-Sequence Models Can Directly Translate Foreign Speech Ron J. Weiss J. Chorowski Navdeep Jaitly Yonghui Wu Zhiwen Chen 79 344 0 24 Mar 2017