Screen2Vec: Semantic Embedding of GUI Screens and GUI Components

Screen2Vec: Semantic Embedding of GUI Screens and GUI Components

11 January 2021

Toby Jia-Jun Li

Lindsay Popowski

Tom Michael Mitchell

Papers citing "Screen2Vec: Semantic Embedding of GUI Screens and GUI Components"

17 / 17 papers shown

Title
MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions Yuxuan Liu Hongda Sun Wei Liu Jian Luan Bo Du Rui Yan 63 2 0 24 Feb 2025
MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling Sidong Feng Suyu Ma Han Wang David Kong Chunyang Chen 50 9 0 11 May 2024
AURORA: Navigating UI Tarpits via Automated Neural Screen Understanding Safwat Ali Khan Wenyu Wang Yiran Ren Bin Zhu Jiangfan Shi Alyssa McGowan Wing Lam Kevin Moran 45 1 0 01 Apr 2024
AI Assistance for UX: A Literature Review Through Human-Centered AI Yuwen Lu Yuewen Yang Qinyi Zhao Chengzhi Zhang Toby Jia-Jun Li 37 16 0 08 Feb 2024
Designing with Language: Wireframing UI Design Intent with Generative Large Language Models Sidong Feng Mingyue Yuan Jieshan Chen Zhenchang Xing Chunyang Chen AI4CE 3DV 24 7 0 12 Dec 2023
Screen Correspondence: Mapping Interchangeable Elements between UIs Jason Wu Amanda Swearngin Xiaoyi Zhang Jeffrey Nichols Jeffrey P. Bigham 48 7 0 20 Jan 2023
Enabling Conversational Interaction with Mobile UI using Large Language Models Bryan Wang Gang Li Yang Li 188 132 0 18 Sep 2022
Avgust: Automating Usage-Based Test Generation from Videos of App Executions Yixue Zhao Saghar Talebipour Kesina Baral Hyojae Park Leon Yee Safwat Ali Khan Yuriy Brun N. Medvidović Kevin Moran 28 11 0 06 Sep 2022
Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis E. Schoop Xin Zhou Gang Li Zhourong Chen Björn Hartmann Yang Li HAI FAtt 37 32 0 05 Apr 2022
Jury Learning: Integrating Dissenting Voices into Machine Learning Models Mitchell L. Gordon Michelle S. Lam J. Park Kayur Patel Jeffrey T. Hancock Tatsunori Hashimoto Michael S. Bernstein 32 146 0 07 Feb 2022
A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility Andrea Burns Deniz Arsan Sanjna Agrawal Ranjitha Kumar Kate Saenko Bryan A. Plummer 52 59 0 04 Feb 2022
Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale Gang Li Gilles Baechler Manuel Tragut Yang Li 29 49 0 11 Jan 2022
VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling Yang Li Gang Li Xin Zhou Mostafa Dehghani A. Gritsenko MLLM 45 35 0 10 Dec 2021
Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models Forrest Huang Gang Li Xin Zhou John F. Canny Yang Li DiffM 33 19 0 14 Oct 2021
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning Bryan Wang Gang Li Xin Zhou Zhourong Chen Tovi Grossman Yang Li 172 154 0 07 Aug 2021
Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments Andrea Burns Deniz Arsan Sanjna Agrawal Ranjitha Kumar Kate Saenko Bryan A. Plummer LRM 31 21 0 17 Apr 2021
Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen G. Corrado J. Dean 3DV 331 31,297 0 16 Jan 2013