Dynamic Planning for LLM-based Graphical User Interface Automation

v1v2 (latest)

Dynamic Planning for LLM-based Graphical User Interface Automation

1 October 2024

Shaoqing Zhang

Zhuosheng Zhang

Xinbei Ma

Muyun Yang

Tiejun Zhao

Min Zhang

ArXiv (abs)PDF HTML Github (4★)

Papers citing "Dynamic Planning for LLM-based Graphical User Interface Automation"

16 / 16 papers shown

Title
Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agents Pengzhou Cheng Haowen Hu Zheng Wu Zongru Wu Tianjie Ju Zhuosheng Zhang Zhuosheng Zhang LLMAG AAML 89 0 0 20 May 2025
The Power of Personality: A Human Simulation Perspective to Investigate Large Language Model Agents Yifan Duan Yihong Tang Xuefeng Bai Kehai Chen Junlin Li Min Zhang LLMAG 513 2 0 28 Feb 2025
Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective M. Zhong Chen Zhang Yikun Lei Xikai Liu Yan Gao Yao Hu Kehai Chen Min Zhang 93 5 0 19 Jun 2024
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs Keen You Haotian Zhang E. Schoop Floris Weers Amanda Swearngin Jeffrey Nichols Yinfei Yang Zhe Gan MLLM 96 93 0 08 Apr 2024
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation An Yan Zhengyuan Yang Wanrong Zhu Kevin Qinghong Lin Linjie Li ... Yiwu Zhong Julian McAuley Jianfeng Gao Zicheng Liu Lijuan Wang LLMAG LM&Ro 138 111 0 13 Nov 2023
The Rise and Potential of Large Language Model Based Agents: A Survey Zhiheng Xi Wenxiang Chen Xin Guo Wei He Yiwen Ding ... Wenjuan Qin Yongyan Zheng Xipeng Qiu Xuanjing Huan Tao Gui LM&MA LM&Ro 3DV AI4CE 123 956 0 14 Sep 2023
Reasoning with Language Model is Planning with World Model Shibo Hao Yi Gu Haodi Ma Joshua Jiahua Hong Zhen Wang D. Wang Zhiting Hu ReLM LRM LLMAG 156 600 0 24 May 2023
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing Zhibin Gou Zhihong Shao Yeyun Gong Yelong Shen Yujiu Yang Nan Duan Weizhu Chen KELM LRM 87 396 0 19 May 2023
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face Yongliang Shen Kaitao Song Xu Tan Dongsheng Li Weiming Lu Yueting Zhuang MLLM 125 911 0 30 Mar 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,748 0 15 Mar 2023
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models Chan Hee Song Jiaman Wu Clay Washington Brian M Sadler Wei-Lun Chao Yu-Chuan Su LLMAG LM&Ro 138 422 0 08 Dec 2022
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks Wenhu Chen Xueguang Ma Xinyi Wang William W. Cohen ReLM ReCod LRM 171 825 0 22 Nov 2022
Towards Better Semantic Understanding of Mobile Interfaces Srinivas Sunkara Maria Wang Lijuan Liu Gilles Baechler Yu-Chung Hsiao Jindong Chen Chen Abhanshu Sharma James Stout 76 24 0 06 Oct 2022
ReAct: Synergizing Reasoning and Acting in Language Models Shunyu Yao Jeffrey Zhao Dian Yu Nan Du Izhak Shafran Karthik Narasimhan Yuan Cao LLMAG ReLM LRM 439 2,976 0 06 Oct 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 529 6,293 0 05 Apr 2022
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks Nils Reimers Iryna Gurevych 1.3K 12,316 0 27 Aug 2019