VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

12 July 2023

Wenlong Huang

Chen Wang

Ruohan Zhang

Yunzhu Li

Jiajun Wu

Li Fei-Fei

LM&Ro

ArXiv PDF HTML

Papers citing "VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models"

50 / 160 papers shown

Title
Emergent Abilities of Large Language Models Jason W. Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph ... Tatsunori Hashimoto Oriol Vinyals Percy Liang J. Dean W. Fedus ELM ReLM LRM 267 2,468 0 15 Jun 2022
Large Language Models are Zero-Shot Reasoners Takeshi Kojima S. Gu Machel Reid Yutaka Matsuo Yusuke Iwasawa ReLM LRM 488 4,077 0 24 May 2022
Generalizable Task Planning through Representation Pretraining Chen Wang Danfei Xu Li Fei-Fei LM&Ro 98 29 0 16 May 2022
Simple Open-Vocabulary Object Detection with Vision Transformers Matthias Minderer A. Gritsenko Austin Stone Maxim Neumann Dirk Weissenborn ... Zhuoran Shen Tianlin Li Xiaohua Zhai Thomas Kipf N. Houlsby ObjD CLIP VLM ViT OCL 81 312 0 12 May 2022
Motion Planning around Obstacles with Convex Optimization Tobia Marcucci Mark E. Petersen D. V. Wrangel Russ Tedrake 63 136 0 09 May 2022
Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? Yuchen Cui S. Niekum Abhi Gupta Vikash Kumar Aravind Rajeswaran LM&Ro 75 77 0 23 Apr 2022
What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data Oier Mees Lukás Hermann Wolfram Burgard LM&Ro 87 153 0 13 Apr 2022
Correcting Robot Plans with Natural Language Feedback Pratyusha Sharma Balakumar Sundaralingam Valts Blukis Chris Paxton Tucker Hermans Antonio Torralba Jacob Andreas Dieter Fox 3DV LM&Ro 54 93 0 11 Apr 2022
Semantic Exploration from Language Abstractions and Pretrained Representations Allison C. Tam Neil C. Rabinowitz Andrew Kyle Lampinen Nicholas A. Roy Stephanie C. Y. Chan D. Strouse Jane X. Wang Andrea Banino Felix Hill LM&Ro 84 70 0 08 Apr 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 441 6,222 0 05 Apr 2022
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances Michael Ahn Anthony Brohan Noah Brown Yevgen Chebotar Omar Cortes ... Ted Xiao Peng Xu Sichun Xu Mengyuan Yan Andy Zeng LM&Ro 165 1,951 0 04 Apr 2022
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language Andy Zeng Maria Attarian Brian Ichter K. Choromanski Adrian S. Wong ... Michael S. Ryoo Vikas Sindhwani Johnny Lee Vincent Vanhoucke Peter R. Florence ReLM LRM 137 582 0 01 Apr 2022
R3M: A Universal Visual Representation for Robot Manipulation Suraj Nair Aravind Rajeswaran Vikash Kumar Chelsea Finn Abhi Gupta LM&Ro 71 570 0 23 Mar 2022
Sequence-of-Constraints MPC: Reactive Timing-Optimal Control of Sequential Manipulation Marc Toussaint Jason Harris Jung-Su Ha Danny Driess Wolfgang Hönig 38 27 0 10 Mar 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 806 12,893 0 04 Mar 2022
Improving Intrinsic Exploration with Language Abstractions Jesse Mu Victor Zhong Roberta Raileanu Minqi Jiang Noah D. Goodman Tim Rocktaschel Edward Grefenstette 123 66 0 17 Feb 2022
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning Eric Jang A. Irpan Mohi Khansari Daniel Kappler F. Ebert Corey Lynch Sergey Levine Chelsea Finn LM&Ro 246 538 0 04 Feb 2022
Pre-Trained Language Models for Interactive Decision-Making Shuang Li Xavier Puig Chris Paxton Yilun Du Clinton Jia Wang ... Anima Anandkumar Jacob Andreas Igor Mordatch Antonio Torralba Yuke Zhu LM&Ro 98 258 0 03 Feb 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 752 9,330 0 28 Jan 2022
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents Wenlong Huang Pieter Abbeel Deepak Pathak Igor Mordatch LM&Ro 81 1,102 0 18 Jan 2022
StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects Weiyu Liu Chris Paxton Tucker Hermans Dieter Fox 67 93 0 19 Oct 2021
Ego4D: Around the World in 3,000 Hours of Egocentric Video Kristen Grauman Andrew Westbury Eugene Byrne Zachary Chavis Antonino Furnari ... Mike Zheng Shou Antonio Torralba Lorenzo Torresani Mingfei Yan Jitendra Malik EgoV 365 1,086 0 13 Oct 2021
Skill Induction and Planning with Latent Language Pratyusha Sharma Antonio Torralba Jacob Andreas LM&Ro 232 110 0 04 Oct 2021
CLIPort: What and Where Pathways for Robotic Manipulation Mohit Shridhar Lucas Manuelli Dieter Fox LM&Ro 98 650 0 24 Sep 2021
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation Suraj Nair E. Mitchell Kevin Chen Brian Ichter Silvio Savarese Chelsea Finn LM&Ro OffRL 88 156 0 02 Sep 2021
MERLOT: Multimodal Neural Script Knowledge Models Rowan Zellers Ximing Lu Jack Hessel Youngjae Yu J. S. Park Jize Cao Ali Farhadi Yejin Choi VLM LRM 95 381 0 04 Jun 2021
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World Rowan Zellers Ari Holtzman Matthew E. Peters Roozbeh Mottaghi Aniruddha Kembhavi Ali Farhadi Yejin Choi 69 69 0 01 Jun 2021
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation Xiuye Gu Nayeon Lee Weicheng Kuo Huayu Chen VLM ObjD 265 915 0 28 Apr 2021
MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding Aishwarya Kamath Mannat Singh Yann LeCun Gabriel Synnaeve Ishan Misra Nicolas Carion ObjD VLM 163 881 0 26 Apr 2021
Language Models are Few-Shot Butlers Vincent Micheli Franccois Fleuret 42 33 0 16 Apr 2021
Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation Mayank Mittal David Hoeller Farbod Farshidian Marco Hutter Animesh Garg 49 88 0 18 Mar 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 856 29,341 0 26 Feb 2021
Where2Act: From Pixels to Actions for Articulated 3D Objects Kaichun Mo Leonidas Guibas Mustafa Mukadam Abhinav Gupta Shubham Tulsiani 193 185 0 07 Jan 2021
Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following Valts Blukis Ross A. Knepper Yoav Artzi LM&Ro 43 33 0 14 Nov 2020
Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions Peter Alexander Jansen LM&Ro 25 52 0 29 Sep 2020
Array Programming with NumPy Charles R. Harris K. Millman S. Walt R. Gommers Pauli Virtanen ... Tyler Reddy Warren Weckesser Hameer Abbasi C. Gohlke T. Oliphant 147 14,915 0 18 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 721 41,894 0 28 May 2020
Language Conditioned Imitation Learning over Unstructured Data Corey Lynch P. Sermanet LM&Ro 66 251 0 15 May 2020
Unsupervised Commonsense Question Answering with Self-Talk Vered Shwartz Peter West Ronan Le Bras Chandra Bhagavatula Yejin Choi ReLM SSL AI4MH LRM 61 262 0 11 Apr 2020
SAPIEN: A SimulAted Part-based Interactive ENvironment Fanbo Xiang Yuzhe Qin Kaichun Mo Yikuan Xia Hao Zhu ... He Wang Li Yi Angel X. Chang Leonidas Guibas Hao Su 270 504 0 19 Mar 2020
Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration Cédric Colas Tristan Karch Nicolas Lair Jean-Michel Dussoux Clément Moulin-Frier Peter Ford Dominey Pierre-Yves Oudeyer LM&Ro LLMAG LRM 54 121 0 21 Feb 2020
RLBench: The Robot Learning Benchmark & Learning Environment Stephen James Z. Ma David Rovick Arrojo Andrew J. Davison SSL VLM OffRL 100 550 0 26 Sep 2019
Deep Dynamics Models for Learning Dexterous Manipulation Anusha Nagabandi K. Konolige Sergey Levine Vikash Kumar 216 414 0 25 Sep 2019
Language as an Abstraction for Hierarchical Deep Reinforcement Learning Yiding Jiang S. Gu Kevin Patrick Murphy Chelsea Finn OffRL 50 225 0 18 Jun 2019
A Survey of Reinforcement Learning Informed by Natural Language Jelena Luketina Nantas Nardelli Gregory Farquhar Jakob N. Foerster Jacob Andreas Edward Grefenstette Shimon Whiteson Tim Rocktaschel LM&Ro KELM OffRL LRM 70 282 0 10 Jun 2019
DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions Zhenjia Xu Jiajun Wu Andy Zeng J. Tenenbaum Shuran Song AI4CE 62 114 0 10 Jun 2019
Differentiable MPC for End-to-end Planning and Control Brandon Amos I. D. Rodriguez Jacob Sacks Byron Boots J. Zico Kolter 67 371 0 31 Oct 2018
Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids Yunzhu Li Jiajun Wu Russ Tedrake J. Tenenbaum Antonio Torralba PINN AI4CE 69 397 0 03 Oct 2018
Graph networks as learnable physics engines for inference and control Alvaro Sanchez-Gonzalez N. Heess Jost Tobias Springenberg J. Merel Martin Riedmiller R. Hadsell Peter W. Battaglia GNN AI4CE PINN OCL 196 599 0 04 Jun 2018
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset Dima Damen Hazel Doughty G. Farinella Sanja Fidler Antonino Furnari ... Davide Moltisanti Jonathan Munro Toby Perrett Will Price Michael Wray EgoV 111 1,027 0 08 Apr 2018