Enabling Conversational Interaction with Mobile UI using Large Language Models

18 September 2022

Papers citing "Enabling Conversational Interaction with Mobile UI using Large Language Models"

23 / 23 papers shown

Title
MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions Yuxuan Liu Hongda Sun Wei Liu Jian Luan Bo Du Rui Yan 55 2 0 24 Feb 2025
Classroom Simulacra: Building Contextual Student Generative Agents in Online Education for Learning Behavioral Simulation Songlin Xu Hao-Ning Wen Hongyi Pan Dallas Dominguez Dongyin Hu Xinyu Zhang 70 1 0 04 Feb 2025
Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review Rock Yuren Pang Hope Schroeder Kynnedy Simone Smith Solon Barocas Ziang Xiao Emily Tseng Danielle Bragg 77 3 0 22 Jan 2025
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents Christopher Rawles Sarah Clinckemaillie Yifan Chang Jonathan Waltz Gabrielle Lau ... Daniel Toyama Robert Berry Divya Tyamagundlu Timothy Lillicrap Oriana Riva LLMAG 69 44 0 23 May 2024
MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling Sidong Feng Suyu Ma Han Wang David Kong Chunyang Chen 39 9 0 11 May 2024
A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration Jie Ying Gao Simret Araya Gebreegziabher K. T. W. Choo T. Li S. Perrault Thomas W. Malone 31 30 0 30 Mar 2024
LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing Bryan Wang Yuliang Li Zhaoyang Lv Haijun Xia Yan Xu Raj Sodhi 35 39 0 15 Feb 2024
Understanding the Role of Large Language Models in Personalizing and Scaffolding Strategies to Combat Academic Procrastination Ananya Bhattacharjee Yuchen Zeng Sarah Yi Xu Dana Kulzhabayeva Minyi Ma ... A. Mariakakis Mary P Czerwinski Anastasia Kuzminykh Michael Liut Joseph Jay Williams AI4Ed 19 25 0 21 Dec 2023
ChoiceMates: Supporting Unfamiliar Online Decision-Making with Multi-Agent Conversational Interactions Jeongeon Park Bryan Min Xiaojuan Ma Juho Kim Xiaojuan Ma Juho Kim 51 15 0 02 Oct 2023
PlanFitting: Personalized Exercise Planning with Large Language Model-driven Conversational Agent Donghoon Shin Gary Hsieh Young-Ho Kim 22 3 0 22 Sep 2023
Android in the Wild: A Large-Scale Dataset for Android Device Control Christopher Rawles Alice Li Daniel Rodriguez Oriana Riva Timothy Lillicrap LM&Ro 28 139 0 19 Jul 2023
Towards Digital Nature: Bridging the Gap between Turing Machine Objects and Linguistic Objects in LLMMs for Universal Interaction of Object-Oriented Descriptions Yoichi Ochiai Naruya Kondo Tatsuki Fushimi AI4CE 40 3 0 10 Apr 2023
MenuCraft: Interactive Menu System Design with Large Language Models Amir Hossein Kargaran Nafiseh Nikeghbal Abbas Heydarnoori Hinrich Schütze LLMAG 28 4 0 08 Mar 2023
Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data Jing Wei Sungdong Kim Hyunhoon Jung Young-Ho Kim 25 82 0 14 Jan 2023
Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments Yu Gu Xiang Deng Yu-Chuan Su LLMAG 42 52 0 19 Dec 2022
ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots Yu-Chung Hsiao Fedir Zubach Maria Wang Jindong Chen Victor Carbune Jason Lin Maria Wang Yun Zhu Jindong Chen RALM 157 25 0 16 Sep 2022
Large Language Models are Zero-Shot Reasoners Takeshi Kojima S. Gu Machel Reid Yutaka Matsuo Yusuke Iwasawa ReLM LRM 325 4,077 0 24 May 2022
Can Foundation Models Wrangle Your Data? A. Narayan Ines Chami Laurel J. Orr Simran Arora Christopher Ré LMTD AI4CE 181 214 0 20 May 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 367 8,495 0 28 Jan 2022
Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots Jason Wu Xiaoyi Zhang Jeffrey Nichols Jeffrey P. Bigham 3DV 163 71 0 17 Sep 2021
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning Bryan Wang Gang Li Xin Zhou Zhourong Chen Tovi Grossman Yang Li 167 152 0 07 Aug 2021
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels Xiaoyi Zhang Lilian de Greef Amanda Swearngin Samuel White Kyle I. Murray ... Jeffrey Nichols Jason Wu Chris Fleizach Aaron Everitt Jeffrey P. Bigham 202 167 0 13 Jan 2021
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 280 2,015 0 28 Jul 2020