Affordance-Guided Reinforcement Learning via Visual Prompting

14 July 2024

Papers citing "Affordance-Guided Reinforcement Learning via Visual Prompting"

50 / 63 papers shown

Title
Robotic Visual Instruction Yuchen Li Ziyang Gong Haoyang Li Xiaoqi Huang Haolan Kang Guangping Bai Xianzheng Ma LM&Ro 108 0 0 01 May 2025
TopV-Nav: Unlocking the Top-View Spatial Reasoning Potential of MLLM for Zero-shot Object Navigation Linqing Zhong Chen Gao Zihan Ding Yue Liao Si Liu Shifeng Zhang Xu Zhou Si Liu LRM 137 5 0 25 Nov 2024
GPT-4o System Card OpenAI OpenAI : Aaron Hurst Adam Lerer Adam P. Goucher ... Yuchen He Yuchen Zhang Yujia Jin Yunxing Dai Yury Malkov MLLM 136 750 0 25 Oct 2024
Open-World Reinforcement Learning over Long Short-Term Imagination Jiajian Li Q. Wang Yunbo Wang Xin Jin Yang Li Wenjun Zeng Xiaokang Yang OCL VLM 112 2 0 04 Oct 2024
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation Wenlong Huang Chen Wang Yongqian Li Ruohan Zhang Li Fei-Fei 99 101 0 03 Sep 2024
SAM 2: Segment Anything in Images and Videos Nikhila Ravi Valentin Gabeur Yuan-Ting Hu Ronghang Hu Chaitanya K. Ryali ... Nicolas Carion Chao-Yuan Wu Ross B. Girshick Piotr Dollár Christoph Feichtenhofer VLM MLLM 92 796 0 01 Aug 2024
3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination Jianing Yang Xuweiyi Chen Nikhil Madaan Madhavan Iyengar Shengyi Qian David Fouhey Joyce Chai 3DV 100 11 0 07 Jun 2024
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs Soroush Nasiriany Fei Xia Wenhao Yu Ted Xiao Jacky Liang ... Karol Hausman N. Heess Chelsea Finn Sergey Levine Brian Ichter LM&Ro LRM 52 98 0 12 Feb 2024
RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback Yufei Wang Zhanyi Sun Jesse Zhang Zhou Xian Erdem Biyik David Held Zackory M. Erickson VLM 65 55 0 06 Feb 2024
Adaptive Mobile Manipulation for Articulated Objects In the Open World Haoyu Xiong Russell Mendonca Kenneth Shaw Deepak Pathak 64 41 0 25 Jan 2024
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks Tianhe Ren Shilong Liu Ailing Zeng Jing Lin Kunchang Li ... Feng Li Jie Yang Hongyang Li Qing Jiang Lei Zhang VLM 85 419 0 25 Jan 2024
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities Boyuan Chen Zhuo Xu Sean Kirmani Brian Ichter Danny Driess Pete Florence Dorsa Sadigh Leonidas Guibas Fei Xia LRM ReLM 67 231 0 22 Jan 2024
Chat-3D v2: Bridging 3D Scene and Large Language Models with Object Identifiers Haifeng Huang Zehan Wang Rongjie Huang Luping Liu Xize Cheng Yang Zhao Tao Jin Zhou Zhao 74 12 0 13 Dec 2023
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches Jiayuan Gu Sean Kirmani Paul Wohlhart Yao Lu Montse Gonzalez Arenas ... Hao Su Karol Hausman Chelsea Finn Q. Vuong Ted Xiao 53 69 0 03 Nov 2023
Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback Max Balsells M. Torné Zihan Wang Samedh Desai Pulkit Agrawal Abhishek Gupta 69 10 0 31 Oct 2023
Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning Jingyun Yang Max Sobol Mark Brandon Vu Archit Sharma Jeannette Bohg Chelsea Finn OffRL OnRL 62 22 0 23 Oct 2023
Eureka: Human-Level Reward Design via Coding Large Language Models Yecheng Jason Ma William Liang Guanzhi Wang De-An Huang Osbert Bastani Dinesh Jayaraman Yuke Zhu Linxi Fan A. Anandkumar 57 304 0 19 Oct 2023
The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision) Zhengyuan Yang Linjie Li Kevin Qinghong Lin Jianfeng Wang Chung-Ching Lin Nasim Shakouri Mahmoudabadi Lijuan Wang LM&MA 49 619 0 29 Sep 2023
Prompt a Robot to Walk with Large Language Models Yen-Jen Wang Bike Zhang Jianyu Chen Koushil Sreenath LM&Ro LLMAG 74 50 0 18 Sep 2023
BridgeData V2: A Dataset for Robot Learning at Scale Homer Walke Kevin Black Abraham Lee Moo Jin Kim Maximilian Du ... Andre Wang He Vivek Myers Kuan Fang Chelsea Finn Sergey Levine 54 219 0 24 Aug 2023
Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes Zehan Wang Haifeng Huang Yang Zhao Ziang Zhang Zhou Zhao 77 67 0 17 Aug 2023
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control Anthony Brohan Noah Brown Justice Carbajal Yevgen Chebotar Xi Chen ... Ted Xiao Peng Xu Sichun Xu Tianhe Yu Brianna Zitkovich LM&Ro LRM 91 1,172 0 28 Jul 2023
CoTracker: It is Better to Track Together Nikita Karaev Ignacio Rocco Benjamin Graham Natalia Neverova Andrea Vedaldi Christian Rupprecht VOT ViT 85 252 0 14 Jul 2023
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models Wenlong Huang Chen Wang Ruohan Zhang Yunzhu Li Jiajun Wu Li Fei-Fei LM&Ro 63 498 0 12 Jul 2023
Large Language Models as General Pattern Machines Suvir Mirchandani F. Xia Peter R. Florence Brian Ichter Danny Driess Montse Gonzalez Arenas Kanishka Rao Dorsa Sadigh Andy Zeng LLMAG 91 191 0 10 Jul 2023
Language to Rewards for Robotic Skill Synthesis Wenhao Yu Nimrod Gileadi Chuyuan Fu Sean Kirmani Kuang-Huei Lee ... N. Heess Dorsa Sadigh Jie Tan Yuval Tassa F. Xia LM&Ro 77 276 0 14 Jun 2023
Generalized Planning in PDDL Domains with Pretrained Large Language Models Tom Silver Soham Dan Kavitha Srinivas J. Tenenbaum L. Kaelbling Michael Katz ELM LRM 76 125 0 18 May 2023
StarCoder: may the source be with you! Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov ... Sean M. Hughes Thomas Wolf Arjun Guha Leandro von Werra H. D. Vries 88 740 0 09 May 2023
Text2Motion: From Natural Language Instructions to Feasible Plans Kevin Qinghong Lin Christopher Agia Toki Migimatsu Marco Pavone Jeannette Bohg LM&Ro 59 272 0 21 Mar 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 890 13,788 0 15 Mar 2023
Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning Mitsuhiko Nakamoto Yuexiang Zhai Anika Singh Max Sobol Mark Yi-An Ma Chelsea Finn Aviral Kumar Sergey Levine OffRL OnRL 162 115 0 09 Mar 2023
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models Jiarui Xu Sifei Liu Arash Vahdat Wonmin Byeon Xiaolong Wang Shalini De Mello VLM 254 327 0 08 Mar 2023
PaLM-E: An Embodied Multimodal Language Model Danny Driess F. Xia Mehdi S. M. Sajjadi Corey Lynch Aakanksha Chowdhery ... Marc Toussaint Klaus Greff Andy Zeng Igor Mordatch Peter R. Florence LM&Ro 62 1,594 0 06 Mar 2023
Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning Archit Sharma Ahmed M. Ahmed Rehaan Ahmad Chelsea Finn SSL 89 17 0 02 Mar 2023
Guiding Pretraining in Reinforcement Learning with Large Language Models Yuqing Du Olivia Watkins Zihan Wang Cédric Colas Trevor Darrell Pieter Abbeel Abhishek Gupta Jacob Andreas LM&Ro 54 177 0 13 Feb 2023
PIVOT: Prompting for Video Continual Learning Andrés Villa Juan Carlos León Alcázar Motasem Alfarra Kumail Alhamoud J. Hurtado Fabian Caba Heilbron Alvaro Soto Guohao Li VLM CLL 84 46 0 09 Dec 2022
CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation Vishnu Sashank Dorbala Gunnar Sigurdsson Robinson Piramuthu Jesse Thomason Gaurav Sukhatme LM&Ro 62 55 0 30 Nov 2022
Visual Language Maps for Robot Navigation Chen Huang Oier Mees Andy Zeng Wolfram Burgard LM&Ro 202 355 0 11 Oct 2022
Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials Aviral Kumar Anika Singh F. Ebert Mitsuhiko Nakamoto Yanlai Yang Chelsea Finn Sergey Levine OffRL OnRL 145 67 0 11 Oct 2022
VIMA: General Robot Manipulation with Multimodal Prompts Yunfan Jiang Agrim Gupta Zichen Zhang Guanzhi Wang Yongqiang Dou Yanjun Chen Li Fei-Fei Anima Anandkumar Yuke Zhu Linxi Fan LM&Ro 75 343 0 06 Oct 2022
Open-vocabulary Queryable Scene Representations for Real World Planning Boyuan Chen F. Xia Brian Ichter Kanishka Rao K. Gopalakrishnan Michael S. Ryoo Austin Stone Daniel Kappler LM&Ro 175 183 0 20 Sep 2022
Inner Monologue: Embodied Reasoning through Planning with Language Models Wenlong Huang F. Xia Ted Xiao Harris Chan Jacky Liang ... Tomas Jackson Linda Luu Sergey Levine Karol Hausman Brian Ichter LLMAG LM&Ro LRM 88 880 0 12 Jul 2022
LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action Dhruv Shah B. Osinski Brian Ichter Sergey Levine LM&Ro 206 447 0 10 Jul 2022
Simple Open-Vocabulary Object Detection with Vision Transformers Matthias Minderer A. Gritsenko Austin Stone Maxim Neumann Dirk Weissenborn ... Zhuoran Shen Tianlin Li Xiaohua Zhai Thomas Kipf N. Houlsby ObjD CLIP VLM ViT OCL 81 310 0 12 May 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 357 6,132 0 05 Apr 2022
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances Michael Ahn Anthony Brohan Noah Brown Yevgen Chebotar Omar Cortes ... Ted Xiao Peng Xu Sichun Xu Mengyuan Yan Andy Zeng LM&Ro 145 1,922 0 04 Apr 2022
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language Andy Zeng Maria Attarian Brian Ichter K. Choromanski Adrian S. Wong ... Michael S. Ryoo Vikas Sindhwani Johnny Lee Vincent Vanhoucke Peter R. Florence ReLM LRM 127 578 0 01 Apr 2022
R3M: A Universal Visual Representation for Robot Manipulation Suraj Nair Aravind Rajeswaran Vikash Kumar Chelsea Finn Abhi Gupta LM&Ro 53 556 0 23 Mar 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong Guosheng Lin MLLM BDL VLM CLIP 467 4,283 0 28 Jan 2022
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents Wenlong Huang Pieter Abbeel Deepak Pathak Igor Mordatch LM&Ro 65 1,078 0 18 Jan 2022