Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins

16 June 2025

Papers citing "Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins"

38 / 38 papers shown

Title
A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewards Shivansh Patel Xinchen Yin Wenlong Huang Shubham Garg H. Nayyeri Li Fei-Fei Svetlana Lazebnik Yongqian Li 150 1 0 12 Feb 2025
Robot Learning with Super-Linear Scaling M. Torné Arhan Jain Jiayi Yuan Vidaaranya Macha Lars L. Ankile Anthony Simeonov Pulkit Agrawal Abhishek Gupta LM&Ro OffRL 109 3 0 02 Dec 2024
DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning Zhenyu Jiang Yuqi Xie K. Lin Zhenjia Xu Weikang Wan Ajay Mandlekar Linxi Fan Yuke Zhu 105 31 0 31 Oct 2024
ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination Xinxin Zhao Wenzhe Cai Likun Tang Teng Wang LM&Ro 66 9 0 13 Oct 2024
Automated Creation of Digital Cousins for Robust Policy Learning Tianyuan Dai Josiah Wong Yunfan Jiang Chen Wang Cem Gokmen Ruohan Zhang Jiajun Wu Li Fei-Fei 82 28 0 09 Oct 2024
Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction Justin Kerr Chung Min Kim Mingxuan Wu Brent Yi Qianqian Wang Ken Goldberg Angjoo Kanazawa 56 17 0 26 Sep 2024
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models Matt Deitke Christopher Clark Sangho Lee Rohun Tripathi Yue Yang ... Noah A. Smith Hannaneh Hajishirzi Ross Girshick Ali Farhadi Aniruddha Kembhavi OSLM VLM 80 14 0 25 Sep 2024
SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting M. N. Qureshi Sparsh Garg Francisco Yandún David Held George Kantor Abhisesh Silwal 3DGS 92 22 0 16 Sep 2024
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation Wenlong Huang Chen Wang Yongqian Li Ruohan Zhang Li Fei-Fei 119 114 0 03 Sep 2024
SAM 2: Segment Anything in Images and Videos Nikhila Ravi Valentin Gabeur Yuan-Ting Hu Ronghang Hu Chaitanya K. Ryali ... Nicolas Carion Chao-Yuan Wu Ross B. Girshick Piotr Dollár Christoph Feichtenhofer VLM MLLM 148 917 0 01 Aug 2024
VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation Wentao Zhao Jiaming Chen Ziyu Meng Donghui Mao Ran Song Wei Zhang 100 11 0 13 Jul 2024
Robotic Control via Embodied Chain-of-Thought Reasoning Michał Zawalski William Chen Karl Pertsch Oier Mees Chelsea Finn Sergey Levine LRM LM&Ro 123 88 0 11 Jul 2024
TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach Weikun Peng Jun Lv Yuwei Zeng Haonan Chen Siheng Zhao Jichen Sun Cewu Lu Lin Shao 60 3 0 03 Jul 2024
OpenVLA: An Open-Source Vision-Language-Action Model Moo Jin Kim Karl Pertsch Siddharth Karamcheti Ted Xiao Ashwin Balakrishna ... Russ Tedrake Dorsa Sadigh Sergey Levine Percy Liang Chelsea Finn LM&Ro VLM 235 527 0 13 Jun 2024
2D Gaussian Splatting for Geometrically Accurate Radiance Fields Binbin Huang Zehao Yu Anpei Chen Andreas Geiger Shenghua Gao 3DGS 177 482 0 26 Mar 2024
Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation M. Torné Anthony Simeonov Zechu Li April Chan Tao Chen Abhishek Gupta Pulkit Agrawal 99 68 0 06 Mar 2024
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs Soroush Nasiriany Fei Xia Wenhao Yu Ted Xiao Jacky Liang ... Karol Hausman N. Heess Chelsea Finn Sergey Levine Brian Ichter LM&Ro LRM 83 112 0 12 Feb 2024
Open X-Embodiment: Robotic Learning Datasets and RT-X Models Open X-Embodiment Collaboration Abby OÑeill Abdul Rehman Abhinav Gupta Abhiram Maddukuri ... Zhuo Xu Zichen Jeff Cui Zichen Zhang Zipeng Fu Zipeng Lin LM&Ro 166 526 0 13 Oct 2023
Physically Grounded Vision-Language Models for Robotic Manipulation Jensen Gao Bidipta Sarkar F. Xia Ted Xiao Jiajun Wu Brian Ichter Anirudha Majumdar Dorsa Sadigh LM&Ro 84 132 0 05 Sep 2023
3D Gaussian Splatting for Real-Time Radiance Field Rendering Bernhard Kerbl Georgios Kopanas Thomas Leimkuehler G. Drettakis 3DGS 231 3,783 0 08 Aug 2023
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control Anthony Brohan Noah Brown Justice Carbajal Yevgen Chebotar Xi Chen ... Ted Xiao Peng Xu Sichun Xu Tianhe Yu Brianna Zitkovich LM&Ro LRM 169 1,285 0 28 Jul 2023
Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation Bokui (William) Shen Ge Yang Alan Yu J. Wong L. Kaelbling Phillip Isola VLM 90 112 0 27 Jul 2023
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models Wenlong Huang Chen Wang Ruohan Zhang Yunzhu Li Jiajun Wu Li Fei-Fei LM&Ro 114 517 0 12 Jul 2023
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought Yao Mu Qinglong Zhang Mengkang Hu Wen Wang Mingyu Ding Jun Jin Bin Wang Jifeng Dai Yu Qiao Ping Luo LM&Ro LRM 86 242 0 24 May 2023
PaLM-E: An Embodied Multimodal Language Model Danny Driess F. Xia Mehdi S. M. Sajjadi Corey Lynch Aakanksha Chowdhery ... Marc Toussaint Klaus Greff Andy Zeng Igor Mordatch Peter R. Florence LM&Ro 117 1,663 0 06 Mar 2023
ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills Jiayuan Gu Fanbo Xiang Xuanlin Li Z. Ling Xiqiang Liu ... Xiao Yuan P. Xie Zhiao Huang Rui Chen Hao Su 109 189 0 09 Feb 2023
Diffusion-based Generation, Optimization, and Planning in 3D Scenes Siyuan Huang Zan Wang Puhao Li Baoxiong Jia Tengyu Liu Yixin Zhu Wei Liang Song-Chun Zhu DiffM 110 217 0 15 Jan 2023
NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction Yiming Wang Qin Han Marc Habermann Kostas Daniilidis Christian Theobalt Lingjie Liu 3DH 72 260 0 10 Dec 2022
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances Michael Ahn Anthony Brohan Noah Brown Yevgen Chebotar Omar Cortes ... Ted Xiao Peng Xu Sichun Xu Mengyuan Yan Andy Zeng LM&Ro 192 1,984 0 04 Apr 2022
Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields Jonathan T. Barron B. Mildenhall Dor Verbin Pratul P. Srinivasan Peter Hedman 196 1,694 0 23 Nov 2021
3D Neural Scene Representations for Visuomotor Control Yunzhu Li Shuang Li Vincent Sitzmann Pulkit Agrawal Antonio Torralba 114 141 0 08 Jul 2021
NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction Peng Wang Lingjie Liu Yuan Liu Christian Theobalt Taku Komura Wenping Wang 104 1,739 0 20 Jun 2021
Model Predictive Robot-Environment Interaction Control for Mobile Manipulation Tasks M. Minniti R. Grandia Kevin Fäh Farbod Farshidian Marco Hutter 58 34 0 08 Jun 2021
UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction Michael Oechsle Songyou Peng Andreas Geiger 111 752 0 20 Apr 2021
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields Jonathan T. Barron B. Mildenhall Matthew Tancik Peter Hedman Ricardo Martín Brualla Pratul P. Srinivasan 121 1,984 0 24 Mar 2021
Feedback MPC for Torque-Controlled Legged Robots R. Grandia Farbod Farshidian René Ranftl Marco Hutter 65 162 0 15 May 2019
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control F. Ebert Chelsea Finn Sudeep Dasari Annie Xie Alex X. Lee Sergey Levine SSL 110 392 0 03 Dec 2018
Deep Visual Foresight for Planning Robot Motion Chelsea Finn Sergey Levine 119 788 0 03 Oct 2016