ScreenAI: A Vision-Language Model for UI and Infographics Understanding

ScreenAI: A Vision-Language Model for UI and Infographics Understanding

7 February 2024

Gilles Baechler

Srinivas Sunkara

Abhanshu Sharma

Papers citing "ScreenAI: A Vision-Language Model for UI and Infographics Understanding"

18 / 18 papers shown

Title
EcoAgent: An Efficient Edge-Cloud Collaborative Multi-Agent Framework for Mobile Automation Biao Yi Xavier Hu Y. Chen Shengyu Zhang Hongxia Yang Fan Wu Fei Wu LLMAG 164 0 0 08 May 2025
AndroidGen: Building an Android Language Agent under Data Scarcity Hanyu Lai Junjie Gao Xiao-Yang Liu Y. Xu S. Zhang Yuxiao Dong Jie Tang LLMAG 74 0 0 27 Apr 2025
SpiritSight Agent: Advanced GUI Agent with One Look Zhiyuan Huang Ziming Cheng Junting Pan Zhaohui Hou Mingjie Zhan LLMAG 101 2 0 05 Mar 2025
MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions Yuxuan Liu Hongda Sun Wei Liu Jian Luan Bo Du Rui Yan 55 2 0 24 Feb 2025
Vision-Language Models for Edge Networks: A Comprehensive Survey Ahmed Sharshar Latif U. Khan Waseem Ullah Mohsen Guizani VLM 70 3 0 11 Feb 2025
VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document Understanding Ofir Abramovich Niv Nayman Sharon Fogel I. Lavi Ron Litman Shahar Tsiper Royee Tichauer Srikar Appalaraju Shai Mazor R. Manmatha VLM 33 3 0 17 Jul 2024
VideoGUI: A Benchmark for GUI Automation from Instructional Videos Kevin Qinghong Lin Linjie Li Difei Gao Qinchen Wu Mingyi Yan Zhengyuan Yang Lijuan Wang Mike Zheng Shou 46 10 0 14 Jun 2024
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs Keen You Haotian Zhang E. Schoop Floris Weers Amanda Swearngin Jeffrey Nichols Yinfei Yang Zhe Gan MLLM 47 82 0 08 Apr 2024
Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs Victor Carbune Hassan Mansoor Fangyu Liu Rahul Aralikatte Gilles Baechler Jindong Chen Abhanshu Sharma ReLM LRM 133 12 0 19 Mar 2024
Android in the Zoo: Chain-of-Action-Thought for GUI Agents Jiwen Zhang Jihao Wu Yihua Teng Minghui Liao Nuo Xu Xiao Xiao Zhongyu Wei Duyu Tang LLMAG LM&Ro 32 50 0 05 Mar 2024
Understanding HTML with Large Language Models Izzeddin Gur Ofir Nachum Yingjie Miao Mustafa Safdari Austin Huang Aakanksha Chowdhery Sharan Narang Noah Fiedel Aleksandra Faust AI4CE 138 70 0 08 Oct 2022
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding Kenton Lee Mandar Joshi Iulia Turc Hexiang Hu Fangyu Liu Julian Martin Eisenschlos Urvashi Khandelwal Peter Shaw Ming-Wei Chang Kristina Toutanova CLIP VLM 160 263 0 07 Oct 2022
ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots Yu-Chung Hsiao Fedir Zubach Maria Wang Jindong Chen Victor Carbune Jason Lin Maria Wang Yun Zhu Jindong Chen RALM 157 25 0 16 Sep 2022
PreSTU: Pre-Training for Scene-Text Understanding Jihyung Kil Soravit Changpinyo Xi Chen Hexiang Hu Sebastian Goodman Wei-Lun Chao Radu Soricut VLM 135 29 0 12 Sep 2022
Pix2seq: A Language Modeling Framework for Object Detection Ting-Li Chen Saurabh Saxena Lala Li David J. Fleet Geoffrey E. Hinton MLLM ViT VLM 241 344 0 22 Sep 2021
Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots Jason Wu Xiaoyi Zhang Jeffrey Nichols Jeffrey P. Bigham 3DV 163 71 0 17 Sep 2021
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning Bryan Wang Gang Li Xin Zhou Zhourong Chen Tovi Grossman Yang Li 167 152 0 07 Aug 2021
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels Xiaoyi Zhang Lilian de Greef Amanda Swearngin Samuel White Kyle I. Murray ... Jeffrey Nichols Jason Wu Chris Fleizach Aaron Everitt Jeffrey P. Bigham 194 167 0 13 Jan 2021