Physically Grounded Vision-Language Models for Robotic Manipulation

5 September 2023

Jiajun Wu

Anirudha Majumdar

Dorsa Sadigh

LM&Ro

ArXiv PDF HTML

Papers citing "Physically Grounded Vision-Language Models for Robotic Manipulation"

30 / 30 papers shown

Title
Symbolically-Guided Visual Plan Inference from Uncurated Video Data Wenyan Yang Ahmet Tikna Yi Zhao Yuying Zhang Luigi Palopoli Marco Roveri Joni Pajarinen VGen 26 0 0 13 May 2025
3D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Tasks V. Bhat Yu-Hsiang Lan Prashanth Krishnamurthy Ramesh Karri Farshad Khorrami 52 0 0 09 May 2025
ReLI: A Language-Agnostic Approach to Human-Robot Interaction Linus Nwankwo Bjoern Ellensohn Ozan Özdenizci Elmar Rueckert LM&Ro 58 0 0 03 May 2025
Dynamic Robot Tool Use with Vision Language Models Noah Trupin Zixing Wang A. H. Qureshi 37 0 0 02 May 2025
Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning Yibo Yan Shen Wang Jiahao Huo Jingheng Ye Zhendong Chu Xuming Hu Philip S. Yu Carla P. Gomes B. Selman Qingsong Wen LRM 127 9 0 05 Feb 2025
InterDyn: Controllable Interactive Dynamics with Video Diffusion Models Rick Akkerman Haiwen Feng M. Black Dimitrios Tzionas Victoria Fernandez-Abrevaya VGen AI4CE 105 3 0 16 Dec 2024
Robust Contact-rich Manipulation through Implicit Motor Adaptation Teng Xue Amirreza Razmjoo Suhan Shetty Sylvain Calinon 102 1 0 16 Dec 2024
Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuning Ji Hyeok Jung Eun Tae Kim S. Kim Joo Ho Lee Bumsoo Kim Buru Chang VLM 209 0 0 24 Nov 2024
CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision Gi-Cheon Kang Junghyun Kim Kyuhwan Shim Jun Ki Lee Byoung-Tak Zhang LM&Ro 107 1 1 01 Nov 2024
Task-oriented Robotic Manipulation with Vision Language Models Nurhan Bulus Guran Hanchi Ren Jingjing Deng Xianghua Xie 29 3 0 21 Oct 2024
Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress Christopher Agia Rohan Sinha Jingyun Yang Zi-ang Cao Rika Antonova Marco Pavone Jeannette Bohg 31 7 0 06 Oct 2024
UniAff: A Unified Representation of Affordances for Tool Usage and Articulation with Vision-Language Models Qiaojun Yu Siyuan Huang Xibin Yuan Zhengkai Jiang Ce Hao ... Junbo Wang Liu Liu Hongsheng Li Peng Gao Cewu Lu 75 3 0 30 Sep 2024
Fast and Accurate Task Planning using Neuro-Symbolic Language Models and Multi-level Goal Decomposition Minseo Kwon Yaesol Kim Young J. Kim 39 3 0 28 Sep 2024
Word2Wave: Language Driven Mission Programming for Efficient Subsea Deployments of Marine Robots Ruo Chen David Blow Adnan Abdullah Md Jahidul Islam 50 1 0 27 Sep 2024
MotIF: Motion Instruction Fine-tuning Minyoung Hwang Joey Hejna Dorsa Sadigh Yonatan Bisk 57 1 0 16 Sep 2024
HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Models V. Bhat Prashanth Krishnamurthy Ramesh Karri Farshad Khorrami 46 4 0 16 Sep 2024
Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models Lynn Chua Badih Ghazi Yangsibo Huang Pritish Kamath Ravi Kumar Pasin Manurangsi Amer Sinha Chulin Xie Chiyuan Zhang 66 1 0 23 Jun 2024
Policy Learning with a Language Bottleneck Megha Srivastava Cédric Colas Dorsa Sadigh Jacob Andreas 43 3 0 07 May 2024
ZeroCAP: Zero-Shot Multi-Robot Context Aware Pattern Formation via Large Language Models Vishnunandan L. N. Venkatesh Byung-Cheol Min LM&Ro 74 2 0 02 Apr 2024
Dialogue with Robots: Proposals for Broadening Participation and Research in the SLIVAR Community Casey Kennington Malihe Alikhani Heather Pon-Barry Katherine Atwell Yonatan Bisk ... Jivko Sinapov Angela Stewart Matthew Stone Stefanie Tellex Tom Williams 61 0 0 01 Apr 2024
VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework Chris Kelly Luhui Hu Bang Yang Yu Tian Deshun Yang Cindy Yang Zaoshan Huang Zihao Li Jiayin Hu Yuexian Zou 37 9 0 14 Mar 2024
Chat with the Environment: Interactive Multimodal Perception Using Large Language Models Xufeng Zhao Mengdi Li C. Weber Muhammad Burhan Hafez S. Wermter LLMAG LM&Ro LRM 107 47 0 14 Mar 2023
Vision-Language Models as Success Detectors Yuqing Du Ksenia Konyushkova Misha Denil A. Raju Jessica Landon Felix Hill Nando de Freitas Serkan Cabi MLLM LRM 91 77 0 13 Mar 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 287 4,261 0 30 Jan 2023
Mind's Eye: Grounded Language Model Reasoning through Simulation Ruibo Liu Jason W. Wei S. Gu Te-Yen Wu Soroush Vosoughi Claire Cui Denny Zhou Andrew M. Dai ReLM LRM 118 79 0 11 Oct 2022
Open-vocabulary Queryable Scene Representations for Real World Planning Boyuan Chen F. Xia Brian Ichter Kanishka Rao K. Gopalakrishnan Michael S. Ryoo Austin Stone Daniel Kappler LM&Ro 146 182 0 20 Sep 2022
LAVIS: A Library for Language-Vision Intelligence Dongxu Li Junnan Li Hung Le Guangsen Wang Silvio Savarese Guosheng Lin VLM 129 51 0 15 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 339 12,003 0 04 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 395 8,559 0 28 Jan 2022
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA Zhengyuan Yang Zhe Gan Jianfeng Wang Xiaowei Hu Yumao Lu Zicheng Liu Lijuan Wang 180 402 0 10 Sep 2021