UINav: A Practical Approach to Train On-Device Automation Agents

v1v2v3v4 (latest)

UINav: A Practical Approach to Train On-Device Automation Agents

15 December 2023

Folawiyo Campbell-Ajala

ArXiv (abs)PDF HTML

Papers citing "UINav: A Practical Approach to Train On-Device Automation Agents"

10 / 10 papers shown

Title
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation An Yan Zhengyuan Yang Wanrong Zhu Kevin Qinghong Lin Linjie Li ... Yiwu Zhong Julian McAuley Jianfeng Gao Zicheng Liu Lijuan Wang LLMAG LM&Ro 123 110 0 13 Nov 2023
Few-Shot Semantic Parsing with Language Models Trained On Code Richard Shin Benjamin Van Durme 63 66 0 16 Dec 2021
Learning UI Navigation through Demonstrations composed of Macro Actions Wei Li LLMAG 60 9 0 16 Oct 2021
UIBert: Learning Generic Multimodal Representations for UI Understanding Chongyang Bai Xiaoxue Zang Ying Xu Srinivas Sunkara Abhinav Rastogi Jindong Chen Blaise Agüera y Arcas 63 94 0 29 Jul 2021
AndroidEnv: A Reinforcement Learning Platform for Android Daniel Toyama P. Hamel Anita Gergely Gheorghe Comanici Amelia Glaese Zafarali Ahmed Tyler Jackson Shibl Mourad Doina Precup VLM SSeg 64 75 0 27 May 2021
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels Xiaoyi Zhang Lilian de Greef Amanda Swearngin Samuel White Kyle I. Murray ... Jeffrey Nichols Jason Wu Chris Fleizach Aaron Everitt Jeffrey P. Bigham 343 171 0 13 Jan 2021
ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces Zecheng He Srinivas Sunkara Xiaoxue Zang Ying Xu Lijuan Liu Nevan Wichers Gabriel Schubiner Ruby B. Lee Jindong Chen Blaise Agüera y Arcas 73 79 0 22 Dec 2020
Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? Jieshan Chen Mulong Xie Zhenchang Xing Chunyang Chen Xiwei Xu Liming Zhu Guoqiang Li OOD 49 148 0 12 Aug 2020
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks Mohit Shridhar Jesse Thomason Daniel Gordon Yonatan Bisk Winson Han Roozbeh Mottaghi Luke Zettlemoyer Dieter Fox LM&Ro 109 770 0 03 Dec 2019
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation Kyunghyun Cho B. V. Merrienboer Çağlar Gülçehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio AIMat 1.0K 23,354 0 03 Jun 2014