Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

7 October 2024

Huan Sun

Yu Su

LM&Ro

LLMAG

ArXiv PDF HTML

Papers citing "Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents"

38 / 38 papers shown

Title
Can Global XAI Methods Reveal Injected Bias in LLMs? SHAP vs Rule Extraction vs RuleSHAP Francesco Sovrano 14 0 0 16 May 2025
Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI Benjamin Raphael Ernhofer Daniil Prokhorov Jannica Langner Dominik Bollmann 39 0 0 09 May 2025
ScaleTrack: Scaling and back-tracking Automated GUI Agents Jing Huang Zhixiong Zeng WenKang Han Yufeng Zhong Liming Zheng Shuai Fu Jingyuan Chen Lin Ma 134 0 0 01 May 2025
Visual Test-time Scaling for GUI Agent Grounding Tiange Luo Lajanugen Logeswaran Justin Johnson Honglak Lee 51 0 0 01 May 2025
UFO2: The Desktop AgentOS Chaoyun Zhang He Huang Chiming Ni J. Mu Si Qin ... Minghua Ma Jian-Guang Lou Qingwei Lin Saravan Rajmohan Dongmei Zhang LLMAG 34 0 0 20 Apr 2025
InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners Yuhang Liu Pengxiang Li C. Xie Xavier Hu Xiaotian Han Shengyu Zhang Hongxia Yang Fei Wu LLMAG LM&Ro LRM AI4CE 72 2 0 19 Apr 2025
TongUI: Building Generalized GUI Agents by Learning from Multimodal Web Tutorials Bofei Zhang Zirui Shang Zhi Gao Wang Zhang Rui Xie Xiaojian Ma Tao Yuan Xinxiao Wu Song-Chun Zhu Qing Li LLMAG 37 1 0 17 Apr 2025
UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis Xinyi Liu Xiaoyi Zhang Ziyun Zhang Yan Lu 39 0 0 15 Apr 2025
Breaking the Data Barrier -- Building GUI Agents Through Task Generalization Junlei Zhang Zichen Ding Chang Ma Zijie Chen Qiushi Sun Zhenzhong Lan Junxian He 138 0 0 14 Apr 2025
GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents Run Luo Lu Wang Wanwei He Xiaobo Xia LLMAG 51 8 0 14 Apr 2025
RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users Suyu Ye Haojun Shi Darren Shih Hyokun Yun Tanya Roosta Tianmin Shu 24 0 0 14 Apr 2025
SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills Boyuan Zheng Michael Y. Fatemi Xiaolong Jin Zhilin Wang Apurva Gandhi ... Yu Gu Jayanth Srinivasa Gaowen Liu Graham Neubig Yu Su CLL 41 1 0 09 Apr 2025
On the Robustness of GUI Grounding Models Against Image Attacks Haoren Zhao Tianyi Chen Zhen Wang AAML 38 1 0 07 Apr 2025
ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use Kaixin Li Ziyang Meng Hongzhan Lin Ziyang Luo Yuchen Tian Jing Ma Zhiyong Huang Tat-Seng Chua 40 9 0 04 Apr 2025
An Illusion of Progress? Assessing the Current State of Web Agents Tianci Xue Weijian Qi Tianneng Shi Chan Hee Song Boyu Gou D. Song Huan Sun Yu Su LLMAG ELM 104 4 1 02 Apr 2025
Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents Saaket Agashe Kyle Wong Vincent Tu Jiachen Yang Ang Li Xin Wang LLMAG 68 1 0 01 Apr 2025
A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models Liangbo Ning Ziran Liang Zhuohang Jiang Haohao Qu Yujuan Ding ... Xiao Wei Shanru Lin Hui Liu Philip S. Yu Qing Li LLMAG LM&Ro 91 6 0 30 Mar 2025
UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning Zhengxi Lu Yuxiang Chai Yaxuan Guo Xi Yin Liang Liu Hao Wang Han Xiao Shuai Ren Guanjing Xiong Yiming Li LLMAG LRM 78 11 0 27 Mar 2025
Boosting Virtual Agent Learning and Reasoning: A Step-wise, Multi-dimensional, and Generalist Reward Model with Benchmark Bingchen Miao Y. Wu Minghe Gao Qifan Yu Wendong Bu Wenqiao Zhang Yunfei Li Siliang Tang Tat-Seng Chua Juncheng Billy Li LLMAG LRM 58 0 0 24 Mar 2025
Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment Gaole Dai Shiqi Jiang Ting Cao Yuanchun Li Yi Yang Rui Tan Mo Li Lili Qiu 48 1 0 20 Mar 2025
UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction Shravan Nayak Xiangru Jian Kevin Qinghong Lin Juan A. Rodriguez Montek Kalsi ... David Vazquez Christopher Pal Perouz Taslakian Spandana Gella Sai Rajeswar 194 0 0 19 Mar 2025
DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agents Yibin Xu Liang Yang Hao Chen Hua Wang Zhi Chen Yaohua Tang 3DV 63 0 0 14 Mar 2025
In-Context Defense in Computer Agents: An Empirical Study Pei Yang Hai Ci Mike Zheng Shou AAML LLMAG 91 0 0 12 Mar 2025
Haste Makes Waste: Evaluating Planning Abilities of LLMs for Efficient and Feasible Multitasking with Time Constraints Between Actions Zirui Wu Xiao Liu Jiayi Li Lingpeng Kong Yansong Feng 42 1 0 04 Mar 2025
Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Tasks Zongru Wu Pengzhou Cheng Zheng Wu Tianjie Ju Zhuosheng Zhang Gongshen Liu LRM 32 1 0 01 Mar 2025
Programming with Pixels: Computer-Use Meets Software Engineering Pranjal Aggarwal Sean Welleck 45 0 0 24 Feb 2025
Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation Yi Yang Ajay Patel Matt Deitke Tanmay Gupta Luca Weihs ... Mark Yatskar Chris Callison-Burch Ranjay Krishna Aniruddha Kembhavi Christopher Clark SyDa 78 2 0 21 Feb 2025
Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents Vardaan Pahuja Yadong Lu Corby Rosset Boyu Gou Arindam Mitra Spencer Whitehead Yu Su Ahmed Awadallah LLMAG LM&Ro Presented at ResearchTrend Connect \| LLMAG on 14 Mar 2025 151 3 1 20 Feb 2025
AppVLM: A Lightweight Vision Language Model for Online App Control Georgios Papoudakis Thomas Coste Zhihao Wu Jianye Hao Jun Wang Kun Shao 57 1 0 10 Feb 2025
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection Y. Liu Pengxiang Li Zishu Wei C. Xie Xueyu Hu Xinchen Xu Shengyu Zhang Xiaotian Han Hongxia Yang Fei Wu LLMAG LRM 53 11 0 08 Jan 2025
PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital World Yanheng He Jiahe Jin Shijie Xia Jiadi Su Runze Fan Haoyang Zou Xiangkun Hu Pengfei Liu LLMAG 41 2 0 23 Dec 2024
Aria-UI: Visual Grounding for GUI Instructions Yuhao Yang Yue Wang Dongxu Li Ziyang Luo Bei Chen Chenyu Huang Junnan Li LM&Ro LLMAG 106 14 0 20 Dec 2024
The BrowserGym Ecosystem for Web Agent Research Thibault Le Sellier De Chezelles Maxime Gasse Alexandre Lacoste Alexandre Drouin Massimo Caccia ... Siva Reddy Quentin Cappart Graham Neubig Ruslan Salakhutdinov Nicolas Chapados LLMAG 109 9 0 06 Dec 2024
ShowUI: One Vision-Language-Action Model for GUI Visual Agent Kevin Qinghong Lin Linjie Li Difei Gao Zheng Yang Shiwei Wu Zechen Bai Weixian Lei Lijuan Wang Mike Zheng Shou LLMAG 74 13 0 26 Nov 2024
The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use Siyuan Hu Mingyu Ouyang Difei Gao Mike Zheng Shou LM&Ro LLMAG 37 15 0 15 Nov 2024
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents Zhiyong Wu Zhenyu Wu Fangzhi Xu Yian Wang Qiushi Sun ... Kanzhi Cheng Zichen Ding L. Chen Paul Pu Liang Yu Qiao 39 31 0 30 Oct 2024
AutoGLM: Autonomous Foundation Agents for GUIs Xiao Liu Bo Qin Dongzhu Liang Guang Dong Hanyu Lai ... Yujia Wang Yongjun Xu Zehan Qi Yuxiao Dong Jie Tang LLMAG 64 12 0 28 Oct 2024
Harnessing Webpage UIs for Text-Rich Visual Understanding Junpeng Liu Tianyue Ou Yifan Song Yuxiao Qu Wai Lam Chenyan Xiong Wenhu Chen Graham Neubig Xiang Yue 82 6 0 17 Oct 2024