AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn

14 June 2023

Papers citing "AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn"

50 / 65 papers shown

Title
RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graph Sameer Malik Moyuru Yamada Ayush Singh Dishank Aggarwal 183 0 0 06 May 2025
VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoT Zhuo Zhi Qiangqiang Wu Minghe shen W. J. Li Yinchuan Li Kun Shao Kaiwen Zhou LLMAG 40 0 0 06 Apr 2025
UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction Shravan Nayak Xiangru Jian Kevin Qinghong Lin Juan A. Rodriguez Montek Kalsi ... David Vazquez Christopher Pal Perouz Taslakian Spandana Gella Sai Rajeswar 230 1 0 19 Mar 2025
VideoMind: A Chain-of-LoRA Agent for Long Video Reasoning Yong Liu Kevin Qinghong Lin C. Chen Mike Zheng Shou LM&Ro LRM 123 0 0 17 Mar 2025
Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering Zeqing Wang Wentao Wan Qiqing Lao Runmeng Chen Minjie Lang Keze Wang Liang Lin Liang Lin LRM 103 3 0 17 Feb 2025
PoAct: Policy and Action Dual-Control Agent for Generalized Applications Guozhi Yuan Yong Liu Jingli Yang Wei Jia Kai Lin Yansong Gao Shan He Zilin Ding Yiming Li LLMAG 43 0 0 13 Jan 2025
ShowUI: One Vision-Language-Action Model for GUI Visual Agent Kevin Qinghong Lin Linjie Li Difei Gao Zhiyong Yang Shiwei Wu Zechen Bai Weixian Lei Lijuan Wang Mike Zheng Shou LLMAG 76 14 0 26 Nov 2024
CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planning Duo Wu Yufei Guo Yuan Meng Yanning Zhang Le Sun Zhi Wang 222 0 0 25 Nov 2024
Foundations and Recent Trends in Multimodal Mobile Agents: A Survey Biao Wu Yanda Li Meng Fang Zirui Song Zhiwei Zhang Yunchao Wei L. Chen LM&Ro LLMAG OffRL AI4TS 44 5 0 04 Nov 2024
Survey of User Interface Design and Interaction Techniques in Generative AI Applications Reuben Luera Ryan A. Rossi Alexa F. Siu Franck Dernoncourt Tong Yu ... Hanieh Salehy Jian Zhao Samyadeep Basu Puneet Mathur Nedim Lipka AI4TS 65 1 0 28 Oct 2024
Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos Md. Mohaiminul Islam Tushar Nagarajan Huiyu Wang Fu-Jen Chu Kris Kitani Gedas Bertasius Xitong Yang 38 2 0 30 Sep 2024
LLM With Tools: A Survey Zhuocheng Shen 43 8 0 24 Sep 2024
Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely Siyun Zhao Yuqing Yang Zilong Wang Zhiyuan He Luna Qiu Lili Qiu SyDa RALM 3DV 46 35 0 23 Sep 2024
From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models Shengsheng Qian Zuyi Zhou Dizhan Xue Bing Wang Changsheng Xu LRM 39 1 0 19 Sep 2024
AppAgent v2: Advanced Agent for Flexible Mobile Interactions Yanda Li Chi Zhang Wanqi Yang Bin-Bin Fu Pei Cheng Xin Chen Ling Chen Yunchao Wei LLMAG LM&Ro 41 11 0 05 Aug 2024
Learning Video Context as Interleaved Multimodal Sequences S. Shao Pengchuan Zhang Y. Li Xide Xia A. Meso Ziteng Gao Jinheng Xie N. Holliman Mike Zheng Shou 49 5 0 31 Jul 2024
KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Models Kemou Jiang Xuan Cai Zhiyong Cui Aoyong Li Yilong Ren Haiyang Yu Hao Yang Daocheng Fu Licheng Wen Pinlong Cai LLMAG 50 226 0 19 Jul 2024
Lifelong Robot Library Learning: Bootstrapping Composable and Generalizable Skills for Embodied Control with Language Models Georgios Tziafas H. Kasaei KELM LM&Ro 49 8 0 26 Jun 2024
Autonomous Agents for Collaborative Task under Information Asymmetry Wei Liu Chenxi Wang Yifei Wang Zihao Xie Rennai Qiu Yufan Dang Zhuoyun Du Weize Chen Cheng Yang Chen Qian LLMAG 44 4 0 21 Jun 2024
GUI Action Narrator: Where and When Did That Action Take Place? Qinchen Wu Difei Gao Kevin Qinghong Lin Zhuoyu Wu Xiangwu Guo Peiran Li Weichen Zhang Hengxu Wang Mike Zheng Shou 45 3 0 19 Jun 2024
DrVideo: Document Retrieval Based Long Video Understanding Ziyu Ma Chenhui Gou Hengcan Shi Bin Sun Shutao Li Hamid Rezatofighi Jianfei Cai VLM 36 13 0 18 Jun 2024
GUICourse: From General Vision Language Models to Versatile GUI Agents Wentong Chen Junbo Cui Jinyi Hu Yujia Qin Junjie Fang ... Yupeng Huo Yuan Yao Yankai Lin Zhiyuan Liu Maosong Sun LLMAG 33 31 0 17 Jun 2024
VideoGUI: A Benchmark for GUI Automation from Instructional Videos Kevin Qinghong Lin Linjie Li Difei Gao Qinchen Wu Mingyi Yan Zhengyuan Yang Lijuan Wang Mike Zheng Shou 46 10 0 14 Jun 2024
MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex Visual Reasoning Somnath Kumar Yash Gadhia T. Ganu A. Nambi LRM 55 1 0 28 May 2024
Tool Learning with Large Language Models: A Survey Changle Qu Sunhao Dai Xiaochi Wei Hengyi Cai Shuaiqiang Wang Dawei Yin Jun Xu Jirong Wen LLMAG 34 83 0 28 May 2024
ReMoDetect: Reward Models Recognize Aligned LLM's Generations Hyunseok Lee Jihoon Tack Jinwoo Shin DeLMO 40 0 0 27 May 2024
UDKAG: Augmenting Large Vision-Language Models with Up-to-Date Knowledge Chuanhao Li Zhen Li Chenchen Jing Shuo Liu Wenqi Shao Yuwei Wu Ping Luo Yu Qiao Kaipeng Zhang ELM 37 3 0 23 May 2024
Human-Centered LLM-Agent User Interface: A Position Paper Daniel Y. Chin Yuxuan Wang Gus Xia LLMAG 35 1 0 19 May 2024
Anywhere: A Multi-Agent Framework for User-Guided, Reliable, and Diverse Foreground-Conditioned Image Generation Tianyidan Xie Rui Ma Qian Wang Xiaoqian Ye Feixuan Liu Ying Tai Zhenyu Zhang Lanjun Wang Zili Yi DiffM MLLM 47 2 0 29 Apr 2024
From Matching to Generation: A Survey on Generative Information Retrieval Xiaoxi Li Jiajie Jin Yujia Zhou Yuyao Zhang Peitian Zhang Yutao Zhu Zhicheng Dou 3DV 84 46 0 23 Apr 2024
What Are Tools Anyway? A Survey from the Language Model Perspective Zhiruo Wang Zhoujun Cheng Hao Zhu Daniel Fried Graham Neubig 68 27 0 18 Mar 2024
m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks Zixian Ma Weikai Huang Jieyu Zhang Tanmay Gupta Ranjay Krishna 55 18 0 17 Mar 2024
VideoAgent: Long-form Video Understanding with Large Language Model as Agent Xiaohan Wang Yuhui Zhang Orr Zohar Serena Yeung-Levy VLM 124 86 0 15 Mar 2024
Online Adaptation of Language Models with a Memory of Amortized Contexts Jihoon Tack Jaehyung Kim Eric Mitchell Jinwoo Shin Yee Whye Teh Jonathan Richard Schwarz KELM 50 18 0 07 Mar 2024
Budget-Constrained Tool Learning with Planning Yuanhang Zheng Peng Li Mingshi Yan Ji Zhang Fei Huang Yang Liu 40 3 0 25 Feb 2024
Large Multimodal Agents: A Survey Junlin Xie Zhihong Chen Ruifei Zhang Xiang Wan Guanbin Li LM&Ro LLMAG 37 39 0 23 Feb 2024
Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding Mirac Suzgun Adam Tauman Kalai KELM LRM LLMAG ReLM 51 65 0 23 Jan 2024
TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks Zhiruo Wang Daniel Fried Graham Neubig 25 19 0 23 Jan 2024
DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent) Zongxin Yang Guikun Chen Xiaodi Li Wenguan Wang Yi Yang LM&Ro LLMAG 69 35 0 16 Jan 2024
Video Understanding with Large Language Models: A Survey Yunlong Tang Jing Bi Siting Xu Luchuan Song Susan Liang ... Feng Zheng Jianguo Zhang Ping Luo Jiebo Luo Chenliang Xu VLM 54 84 0 29 Dec 2023
Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey Jiaxing Huang Jingyi Zhang Kai Jiang Han Qiu Shijian Lu 44 22 0 27 Dec 2023
LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordination Jijia Liu Chao Yu Jiaxuan Gao Yuqing Xie Qingmin Liao Yi Wu Yu Wang LLMAG LM&Ro 92 35 0 23 Dec 2023
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation Difei Gao Lei Ji Zechen Bai Mingyu Ouyang Peiran Li ... Peiyi Wang Xiangwu Guo Hengxu Wang Luowei Zhou Mike Zheng Shou LLMAG 31 21 0 20 Dec 2023
CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update Zhi Gao Yuntao Du Xintong Zhang Xiaojian Ma Wenjuan Han Song-Chun Zhu Qing Li LLMAG VLM 31 21 0 18 Dec 2023
A Survey of Reasoning with Foundation Models Jiankai Sun Chuanyang Zheng E. Xie Zhengying Liu Ruihang Chu ... Xipeng Qiu Yi-Chen Guo Hui Xiong Qun Liu Zhenguo Li ReLM LRM AI4CE 30 76 0 17 Dec 2023
GAIA: a benchmark for General AI Assistants Grégoire Mialon Clémentine Fourrier Craig Swift Thomas Wolf Yann LeCun Thomas Scialom AI4MH ALM ELM RALM 17 141 0 21 Nov 2023
Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions Ziyue Wang Chi Chen Peng Li Yang Liu LRM 23 14 0 20 Nov 2023
How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model Shezheng Song Xiaopeng Li Shasha Li Shan Zhao Jie Yu Jun Ma Xiaoguang Mao Weimin Zhang 71 4 0 10 Nov 2023
Sentiment Analysis through LLM Negotiations Xiaofei Sun Xiaoya Li Shengyu Zhang Shuhe Wang Fei Wu Jiwei Li Tianwei Zhang Guoyin Wang 35 16 0 03 Nov 2023
LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation Kibum Kim Kanghoon Yoon Jaeyeong Jeon Yeonjun In Jinyoung Moon Donghyun Kim Chanyoung Park 31 14 0 16 Oct 2023