v1v2 (latest)

AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn

14 June 2023

Papers citing "AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn"

50 / 60 papers shown

Title
Efficiently Enhancing General Agents With Hierarchical-categorical Memory Changze Qiao Mingming Lu LLMAG 34 0 0 28 May 2025
RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graph Sameer Malik Moyuru Yamada Ayush Singh Dishank Aggarwal 451 0 0 06 May 2025
VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoT Zhuo Zhi Qiangqiang Wu Minghe shen Wenbo Li Yinchuan Li Kun Shao Kaiwen Zhou LLMAG 182 3 0 06 Apr 2025
UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction Shravan Nayak Xiangru Jian Kevin Qinghong Lin Juan A. Rodriguez Montek Kalsi ... David Vazquez Christopher Pal Perouz Taslakian Spandana Gella Sai Rajeswar 515 4 0 19 Mar 2025
VideoMind: A Chain-of-LoRA Agent for Long Video Reasoning Yang Liu Kevin Qinghong Lin C. Chen Mike Zheng Shou LM&Ro LRM 391 6 0 17 Mar 2025
Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering Zeqing Wang Wentao Wan Qiqing Lao Runmeng Chen Minjie Lang Keze Wang Liang Lin Liang Lin LRM 234 3 0 17 Feb 2025
PoAct: Policy and Action Dual-Control Agent for Generalized Applications Guozhi Yuan Yang Liu Jingli Yang Wei Jia Kai Lin Yansong Gao Shan He Zilin Ding Haoyang Li LLMAG 53 0 0 13 Jan 2025
ShowUI: One Vision-Language-Action Model for GUI Visual Agent Kevin Qinghong Lin Linjie Li Difei Gao Zhiyong Yang Shiwei Wu Zechen Bai Weixian Lei Lijuan Wang Mike Zheng Shou LLMAG 148 36 0 26 Nov 2024
CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planning Duo Wu Jiangming Wang Yuan Meng Yanning Zhang Le Sun Zhi Wang 538 0 0 25 Nov 2024
Foundations and Recent Trends in Multimodal Mobile Agents: A Survey Biao Wu Yanda Li Meng Fang Zirui Song Zhiwei Zhang LM&Ro LLMAG OffRL AI4TS 87 7 0 04 Nov 2024
Survey of User Interface Design and Interaction Techniques in Generative AI Applications Reuben Luera Ryan Rossi Alexa F. Siu Franck Dernoncourt Tong Yu ... Hanieh Salehy Jian Zhao Samyadeep Basu Puneet Mathur Nedim Lipka AI4TS 139 1 0 28 Oct 2024
Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos Md. Mohaiminul Islam Tushar Nagarajan Huiyu Wang Fu-Jen Chu Kris Kitani Gedas Bertasius Xitong Yang 76 4 0 30 Sep 2024
LLM With Tools: A Survey Zhuocheng Shen 85 14 0 24 Sep 2024
Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely Siyun Zhao Yuqing Yang Zilong Wang Zhiyuan He Luna Qiu Lili Qiu SyDa RALM 3DV 120 42 0 23 Sep 2024
From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models Shengsheng Qian Zuyi Zhou Dizhan Xue Bing Wang Changsheng Xu LRM 157 2 0 19 Sep 2024
AppAgent v2: Advanced Agent for Flexible Mobile Interactions Yanda Li Chi Zhang Wanqi Yang Bin-Bin Fu Pei Cheng Xin Chen Ling Chen Yunchao Wei LLMAG LM&Ro 116 27 0 05 Aug 2024
Learning Video Context as Interleaved Multimodal Sequences S. Shao Pengchuan Zhang Y. Li Xide Xia A. Meso Ziteng Gao Jinheng Xie N. Holliman Mike Zheng Shou 108 6 0 31 Jul 2024
KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Models Kemou Jiang Xuan Cai Zhiyong Cui Aoyong Li Yilong Ren Haiyang Yu Hao Yang Daocheng Fu Licheng Wen Pinlong Cai LLMAG 100 10 0 19 Jul 2024
Lifelong Robot Library Learning: Bootstrapping Composable and Generalizable Skills for Embodied Control with Language Models Georgios Tziafas Hamidreza Kasaei KELM LM&Ro 98 9 0 26 Jun 2024
Autonomous Agents for Collaborative Task under Information Asymmetry Wei Liu Chenxi Wang Yifei Wang Zihao Xie Rennai Qiu Yufan Dang Zhuoyun Du Weize Chen Cheng Yang Chen Qian LLMAG 100 6 0 21 Jun 2024
GUI Action Narrator: Where and When Did That Action Take Place? Qinchen Wu Difei Gao Kevin Qinghong Lin Zhuoyu Wu Xiangwu Guo Peiran Li Weichen Zhang Hengxu Wang Mike Zheng Shou 105 3 0 19 Jun 2024
DrVideo: Document Retrieval Based Long Video Understanding Ziyu Ma Chenhui Gou Hengcan Shi Bin Sun Shutao Li Hamid Rezatofighi Jianfei Cai VLM 82 15 0 18 Jun 2024
GUICourse: From General Vision Language Models to Versatile GUI Agents Wentong Chen Junbo Cui Jinyi Hu Yujia Qin Junjie Fang ... Yupeng Huo Yuan Yao Yankai Lin Zhiyuan Liu Maosong Sun LLMAG 162 41 0 17 Jun 2024
VideoGUI: A Benchmark for GUI Automation from Instructional Videos Kevin Qinghong Lin Linjie Li Difei Gao Qinchen Wu Mingyi Yan Zhengyuan Yang Lijuan Wang Mike Zheng Shou 117 13 0 14 Jun 2024
Towards Rationality in Language and Multimodal Agents: A Survey Bowen Jiang Yangxinyu Xie Xiaomeng Wang Yuan Yuan Camillo J Taylor Tanwi Mallick Weijie J. Su Camillo J. Taylor Tanwi Mallick LLMAG 91 6 0 01 Jun 2024
MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex Visual Reasoning Somnath Kumar Yash Gadhia T. Ganu A. Nambi LRM 141 4 0 28 May 2024
Tool Learning with Large Language Models: A Survey Changle Qu Sunhao Dai Xiaochi Wei Hengyi Cai Shuaiqiang Wang D. Yin Jun Xu Jirong Wen LLMAG 105 107 0 28 May 2024
ReMoDetect: Reward Models Recognize Aligned LLM's Generations Hyunseok Lee Jihoon Tack Jinwoo Shin DeLMO 65 1 0 27 May 2024
UDKAG: Augmenting Large Vision-Language Models with Up-to-Date Knowledge Chuanhao Li Zhen Li Chenchen Jing Shuo Liu Wenqi Shao Yuwei Wu Ping Luo Yu Qiao Kaipeng Zhang ELM 81 6 0 23 May 2024
Human-Centered LLM-Agent User Interface: A Position Paper Daniel Y. Chin Yuxuan Wang Gus Xia LLMAG 80 2 0 19 May 2024
Anywhere: A Multi-Agent Framework for User-Guided, Reliable, and Diverse Foreground-Conditioned Image Generation Tianyidan Xie Rui Ma Qian Wang Xiaoqian Ye Feixuan Liu Ying Tai Zhenyu Zhang Lanjun Wang Zili Yi DiffM MLLM 106 2 0 29 Apr 2024
From Matching to Generation: A Survey on Generative Information Retrieval Xiaoxi Li Jiajie Jin Yujia Zhou Yuyao Zhang Peitian Zhang Yutao Zhu Zhicheng Dou 3DV 215 61 0 23 Apr 2024
What Are Tools Anyway? A Survey from the Language Model Perspective Zhiruo Wang Zhoujun Cheng Hao Zhu Daniel Fried Graham Neubig 130 33 0 18 Mar 2024
m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks Zixian Ma Weikai Huang Jieyu Zhang Tanmay Gupta Ranjay Krishna 114 22 0 17 Mar 2024
VideoAgent: Long-form Video Understanding with Large Language Model as Agent Xiaohan Wang Yuhui Zhang Orr Zohar Serena Yeung-Levy VLM 209 107 0 15 Mar 2024
Online Adaptation of Language Models with a Memory of Amortized Contexts Jihoon Tack Jaehyung Kim Eric Mitchell Jinwoo Shin Yee Whye Teh Jonathan Richard Schwarz KELM 101 20 0 07 Mar 2024
Budget-Constrained Tool Learning with Planning Yuanhang Zheng Peng Li Mingshi Yan Ji Zhang Fei Huang Yang Liu 136 6 0 25 Feb 2024
Large Multimodal Agents: A Survey Junlin Xie Zhihong Chen Ruifei Zhang Xiang Wan Guanbin Li LM&Ro LLMAG 100 44 0 23 Feb 2024
Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding Mirac Suzgun Adam Tauman Kalai KELM LRM LLMAG ReLM 123 78 0 23 Jan 2024
TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks Zhiruo Wang Daniel Fried Graham Neubig 88 24 0 23 Jan 2024
DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent) Zongxin Yang Guikun Chen Xiaodi Li Wenguan Wang Yi Yang LM&Ro LLMAG 196 41 0 16 Jan 2024
Video Understanding with Large Language Models: A Survey Yunlong Tang Jing Bi Siting Xu Luchuan Song Susan Liang ... Feng Zheng Jianguo Zhang Chenliang Xu Jiebo Luo Chenliang Xu VLM 222 100 0 29 Dec 2023
Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey Jiaxing Huang Jingyi Zhang Kai Jiang Han Qiu Shijian Lu 92 23 0 27 Dec 2023
LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordination Jijia Liu Chao Yu Jiaxuan Gao Yuqing Xie Qingmin Liao Yi Wu Yu Wang LLMAG LM&Ro 178 38 0 23 Dec 2023
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation Difei Gao Lei Ji Zechen Bai Mingyu Ouyang Peiran Li ... Peiyi Wang Xiangwu Guo Hengxu Wang Luowei Zhou Mike Zheng Shou LLMAG 109 24 0 20 Dec 2023
CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update Zhi Gao Yuntao Du Xintong Zhang Xiaojian Ma Wenjuan Han Song-Chun Zhu Qing Li LLMAG VLM 135 25 0 18 Dec 2023
A Survey of Reasoning with Foundation Models Jiankai Sun Chuanyang Zheng Enze Xie Zhengying Liu Ruihang Chu ... Xipeng Qiu Yi-Chen Guo Hui Xiong Qun Liu Zhenguo Li ReLM LRM AI4CE 209 85 0 17 Dec 2023
GAIA: a benchmark for General AI Assistants Grégoire Mialon Clémentine Fourrier Craig Swift Thomas Wolf Yann LeCun Thomas Scialom AI4MH ALM ELM RALM 103 186 0 21 Nov 2023
Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions Ziyue Wang Chi Chen Peng Li Yang Liu LRM 86 16 0 20 Nov 2023
How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model Shangwen Wang Xiaopeng Li Shasha Li Shan Zhao Jie Yu Jun Ma Xiaoguang Mao Weimin Zhang 119 7 0 10 Nov 2023