v1v2v3 (latest)

From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Models

31 December 2024

Leslie Pack Kaelbling

Leslie Kaelbling

Author Contacts:

ashay@mit.edu njk@mit.edu

LM&Ro

ArXiv (abs)PDF HTML

Papers citing "From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Models"

32 / 32 papers shown

Title
Learning Compositional Behaviors from Demonstration and Language Weiyu Liu Neil Nie Ruohan Zhang Jiayuan Mao Jiajun Wu LM&Ro 41 6 0 28 May 2025
Coloring Between the Lines: Personalization in the Null Space of Planning Constraints Tom Silver Rajat Kumar Jenamani Ziang Liu Ben Dodson Tapomayukh Bhattacharjee 53 0 0 21 May 2025
ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models Matteo Merler Nicola Dainese Minttu Alakuijala Giovanni Bonetta Pietro Ferrazzi Yu Tian Bernardo Magnini Pekka Marttinen LM&Ro VLM 95 0 0 19 May 2025
Symbolically-Guided Visual Plan Inference from Uncurated Video Data Wenyan Yang Ahmet Tikna Yi Zhao Yuying Zhang Luigi Palopoli Marco Roveri Joni Pajarinen VGen 48 0 0 13 May 2025
Bilevel Learning for Bilevel Planning Bowen Li Tom Silver Sebastian A. Scherer Alexander G. Gray 232 2 0 12 Feb 2025
Open-World Task and Motion Planning via Vision-Language Model Inferred Constraints Nishanth Kumar F. Ramos Dieter Fox Caelan Reed Garrett Tomás Lozano-Pérez Leslie Pack Kaelbling Caelan Reed Garrett LRM LM&Ro 97 5 0 13 Nov 2024
VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning Yichao Liang Nishanth Kumar Hao Tang Adrian Weller J. Tenenbaum Tom Silver Joao Henriques Kevin Ellis 95 12 0 30 Oct 2024
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation Wenlong Huang Chen Wang Yongqian Li Ruohan Zhang Li Fei-Fei 114 110 0 03 Sep 2024
Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfaction Aidan Curtis Nishanth Kumar Jing Cao Tomás Lozano-Pérez Leslie Pack Kaelbling 76 14 0 08 Jun 2024
InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning Muzhi Han Yifeng Zhu Song-Chun Zhu Yingnian Wu Yuke Zhu 64 27 0 30 May 2024
An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM Wonkyun Kim Changin Choi Wonseok Lee Wonjong Rhee VLM 85 53 0 27 Mar 2024
Practice Makes Perfect: Planning to Learn Skill Parameter Policies Nishanth Kumar Tom Silver Willie McClinton Linfeng Zhao Stephen Proulx Tomás Lozano-Pérez L. Kaelbling Jennifer Barry 91 21 0 22 Feb 2024
Verifiably Following Complex Robot Instructions with Foundation Models Benedict Quartey Eric Rosen Stefanie Tellex George Konidaris LM&Ro 112 13 0 18 Feb 2024
LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks Subbarao Kambhampati Karthik Valmeekam L. Guan Mudit Verma Kaya Stechly Siddhant Bhambri Lucas Saldyt Anil Murthy LRM 118 123 0 02 Feb 2024
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks Tianhe Ren Shilong Liu Ailing Zeng Jing Lin Kunchang Li ... Feng Li Jie Yang Hongyang Li Qing Jiang Lei Zhang VLM 108 430 0 25 Jan 2024
Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis Yafei Hu Quanting Xie Vidhi Jain Jonathan M Francis Jay Patrikar ... Xiaolong Wang Sebastian A. Scherer Z. Kira Fei Xia Yonatan Bisk LM&Ro AI4CE 75 72 0 14 Dec 2023
Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning Yingdong Hu Fanqi Lin Tong Zhang Li Yi Yang Gao LM&Ro 142 120 0 29 Nov 2023
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V Jianwei Yang Hao Zhang Feng Li Xueyan Zou Chun-yue Li Jianfeng Gao MLLM VLM 82 190 0 17 Oct 2023
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models Wenlong Huang Chen Wang Ruohan Zhang Yunzhu Li Jiajun Wu Li Fei-Fei LM&Ro 109 510 0 12 Jul 2023
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion Cheng Chi Zhenjia Xu S. Feng Eric A. Cousineau Yilun Du Benjamin Burchfiel Russ Tedrake Shuran Song 347 1,189 0 07 Mar 2023
Learning Efficient Abstract Planning Models that Choose What to Predict Nishanth Kumar Willie McClinton Rohan Chitnis Tom Silver Tomás Lozano-Pérez L. Kaelbling 85 20 0 16 Aug 2022
Inner Monologue: Embodied Reasoning through Planning with Language Models Wenlong Huang F. Xia Ted Xiao Harris Chan Jacky Liang ... Tomas Jackson Linda Luu Sergey Levine Karol Hausman Brian Ichter LLMAG LM&Ro LRM 129 905 0 12 Jul 2022
Discovering State and Action Abstractions for Generalized Task and Motion Planning Aidan Curtis Tom Silver J. Tenenbaum Tomas Lozano-Perez L. Kaelbling 83 29 0 23 Sep 2021
Learning Neuro-Symbolic Relational Transition Models for Bilevel Planning Rohan Chitnis Tom Silver J. Tenenbaum Tomas Lozano-Perez L. Kaelbling 97 54 0 28 May 2021
Learning Symbolic Operators for Task and Motion Planning Tom Silver Rohan Chitnis J. Tenenbaum L. Kaelbling Tomas Lozano-Perez 97 83 0 28 Feb 2021
Integrated Task and Motion Planning Caelan Reed Garrett Rohan Chitnis Rachel Holladay Beomjoon Kim Tom Silver L. Kaelbling Tomás Lozano-Pérez 100 501 0 02 Oct 2020
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning Abhishek Gupta Vikash Kumar Corey Lynch Sergey Levine Karol Hausman 89 433 0 25 Oct 2019
Learning First-Order Symbolic Representations for Planning from the Structure of the State Space Blai Bonet Hector Geffner NAI 41 53 0 12 Sep 2019
Classical Planning in Deep Latent Space: Bridging the Subsymbolic-Symbolic Boundary Masataro Asai A. Fukunaga 127 172 0 29 Apr 2017
VQA: Visual Question Answering Aishwarya Agrawal Jiasen Lu Stanislaw Antol Margaret Mitchell C. L. Zitnick Dhruv Batra Devi Parikh CoGe 202 5,478 0 03 May 2015
The Fast Downward Planning System M. Helmert 83 1,901 0 27 Sep 2011
PDDL2.1: An Extension to PDDL for Expressing Temporal Planning Domains M. Fox D. Long 82 2,176 0 22 Jun 2011