MUTEX: Learning Unified Policies from Multimodal Task Specifications

25 September 2023

Rutav Shah

Roberto Martín-Martín

Yuke Zhu

OffRL

ArXiv PDF HTML

Papers citing "MUTEX: Learning Unified Policies from Multimodal Task Specifications"

39 / 39 papers shown

Title
Efficient Robotic Policy Learning via Latent Space Backward Planning Dongxiu Liu Haoyi Niu Zhihao Wang Jinliang Zheng Yinan Zheng Zhonghong Ou Jianming Hu Jianxiong Li Xianyuan Zhan 101 0 0 11 May 2025
UniVLA: Learning to Act Anywhere with Task-centric Latent Actions Qingwen Bu Yanting Yang Jisong Cai Shenyuan Gao Guanghui Ren Maoqing Yao Ping Luo Hongyang Li 328 7 0 09 May 2025
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots Nvidia Johan Bjorck Fernando Castañeda Nikita Cherniadev Xingye Da ... Ao Zhang Hao Zhang Yizhou Zhao Ruijie Zheng Yuke Zhu VLM 135 51 0 18 Mar 2025
HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model Jiaming Liu Hao Chen Pengju An Zhuoyang Liu Renrui Zhang ... Chengkai Hou Mengdi Zhao KC alex Zhou Pheng-Ann Heng Shanghang Zhang 129 16 0 13 Mar 2025
RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation Songming Liu Lingxuan Wu Bangguo Li Hengkai Tan Huayu Chen Zhengyi Wang Ke Xu Hang Su Jun Zhu 114 111 0 10 Oct 2024
Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making Vivek Myers Chongyi Zheng Anca Dragan Sergey Levine Benjamin Eysenbach OffRL 97 13 0 24 Jun 2024
Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control Vivek Myers Andre Wang He Kuan Fang Homer Walke Philippe Hansen-Estruch Ching-An Cheng Mihai Jalobeanu Andrey Kolobov Anca Dragan Sergey Levine LM&Ro 52 31 0 30 Jun 2023
Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models Zhiqiu Lin Samuel Yu Zhiyi Kuang Deepak Pathak Deva Ramana VLM 103 113 0 16 Jan 2023
Interactive Language: Talking to Robots in Real Time Corey Lynch Ayzaan Wahid Jonathan Tompson Tianli Ding James Betker Robert Baruch Travis Armstrong Peter R. Florence LM&Ro 77 227 0 12 Oct 2022
VIMA: General Robot Manipulation with Multimodal Prompts Yunfan Jiang Agrim Gupta Zichen Zhang Guanzhi Wang Yongqiang Dou Yanjun Chen Li Fei-Fei Anima Anandkumar Yuke Zhu Linxi Fan LM&Ro 93 350 0 06 Oct 2022
Instruction-driven history-aware policies for robotic manipulations Pierre-Louis Guhur Shizhe Chen Ricardo Garcia Pinel Makarand Tapaswi Ivan Laptev Cordelia Schmid LM&Ro 143 106 0 11 Sep 2022
Multimodal Masked Autoencoders Learn Transferable Representations Xinyang Geng Hao Liu Lisa Lee Dale Schuurams Sergey Levine Pieter Abbeel 64 118 0 27 May 2022
Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? Yuchen Cui S. Niekum Abhi Gupta Vikash Kumar Aravind Rajeswaran LM&Ro 81 77 0 23 Apr 2022
What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data Oier Mees Lukás Hermann Wolfram Burgard LM&Ro 96 153 0 13 Apr 2022
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances Michael Ahn Anthony Brohan Noah Brown Yevgen Chebotar Omar Cortes ... Ted Xiao Peng Xu Sichun Xu Mengyuan Yan Andy Zeng LM&Ro 179 1,954 0 04 Apr 2022
MultiMAE: Multi-modal Multi-task Masked Autoencoders Roman Bachmann David Mizrahi Andrei Atanov Amir Zamir 124 276 0 04 Apr 2022
R3M: A Universal Visual Representation for Robot Manipulation Suraj Nair Aravind Rajeswaran Vikash Kumar Chelsea Finn Abhi Gupta LM&Ro 88 571 0 23 Mar 2022
All in One: Exploring Unified Video-Language Pre-training Alex Jinpeng Wang Yixiao Ge Rui Yan Yuying Ge Xudong Lin Guanyu Cai Jianping Wu Ying Shan Xiaohu Qie Mike Zheng Shou 80 201 0 14 Mar 2022
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning Eric Jang A. Irpan Mohi Khansari Daniel Kappler F. Ebert Corey Lynch Sergey Levine Chelsea Finn LM&Ro 248 538 0 04 Feb 2022
MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound Rowan Zellers Jiasen Lu Ximing Lu Youngjae Yu Yanpeng Zhao Mohammadreza Salehi Aditya Kusupati Jack Hessel Ali Farhadi Yejin Choi 92 212 0 07 Jan 2022
FLAVA: A Foundational Language And Vision Alignment Model Amanpreet Singh Ronghang Hu Vedanuj Goswami Guillaume Couairon Wojciech Galuba Marcus Rohrbach Douwe Kiela CLIP VLM 88 706 0 08 Dec 2021
Grounded Language-Image Pre-training Liunian Harold Li Pengchuan Zhang Haotian Zhang Jianwei Yang Chunyuan Li ... Lu Yuan Lei Zhang Lei Li Kai-Wei Chang Jianfeng Gao ObjD VLM 120 1,061 0 07 Dec 2021
Florence: A New Foundation Model for Computer Vision Lu Yuan Dongdong Chen Yi-Ling Chen Noel Codella Xiyang Dai ... Zhen Xiao Jianwei Yang Michael Zeng Luowei Zhou Pengchuan Zhang VLM 121 904 0 22 Nov 2021
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation Suraj Nair E. Mitchell Kevin Chen Brian Ichter Silvio Savarese Chelsea Finn LM&Ro OffRL 90 156 0 02 Sep 2021
Perceiver IO: A General Architecture for Structured Inputs & Outputs Andrew Jaegle Sebastian Borgeaud Jean-Baptiste Alayrac Carl Doersch Catalin Ionescu ... Olivier J. Hénaff M. Botvinick Andrew Zisserman Oriol Vinyals João Carreira MLLM VLM GNN 64 579 0 30 Jul 2021
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding Hu Xu Gargi Ghosh Po-Yao (Bernie) Huang Prahal Arora Masoumeh Aminzadeh Christoph Feichtenhofer Florian Metze Luke Zettlemoyer 45 132 0 20 May 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 185 1,014 0 04 Mar 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 443 3,842 0 11 Feb 2021
Language Conditioned Imitation Learning over Unstructured Data Corey Lynch P. Sermanet LM&Ro 75 251 0 15 May 2020
Experience Grounds Language Yonatan Bisk Ari Holtzman Jesse Thomason Jacob Andreas Yoshua Bengio ... Angeliki Lazaridou Jonathan May Aleksandr Nisnevich Nicolas Pinto Joseph P. Turian 81 357 0 21 Apr 2020
Learning One-Shot Imitation from Humans without Humans Alessandro Bonardi Stephen James Andrew J. Davison 70 79 0 04 Nov 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 419 20,127 0 23 Oct 2019
LXMERT: Learning Cross-Modality Encoder Representations from Transformers Hao Hao Tan Joey Tianyi Zhou VLM MLLM 237 2,479 0 20 Aug 2019
Connecting Touch and Vision via Cross-Modal Prediction Yunzhu Li Jun-Yan Zhu Russ Tedrake Antonio Torralba 60 136 0 14 Jun 2019
Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks Michelle A. Lee Yuke Zhu K. Srinivasan Parth Shah Silvio Savarese Li Fei-Fei Animesh Garg Jeannette Bohg SSL 71 369 0 24 Oct 2018
Task-Embedded Control Networks for Few-Shot Imitation Learning Stephen James Michael Bloesch Andrew J. Davison 67 136 0 08 Oct 2018
One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning Tianhe Yu Chelsea Finn Annie Xie Sudeep Dasari Tianhao Zhang Pieter Abbeel Sergey Levine 64 360 0 05 Feb 2018
Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration Rouhollah Rahmatizadeh P. Abolghasemi Ladislau Bölöni Sergey Levine 95 256 0 10 Jul 2017
SGDR: Stochastic Gradient Descent with Warm Restarts I. Loshchilov Frank Hutter ODL 327 8,116 0 13 Aug 2016