Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better

29 May 2025

Danny Driess

Jost Tobias Springenberg

Papers citing "Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better"

45 / 45 papers shown

Title
$$π_{0.5}$: a Vision-Language-Action Model with Open-World Generalization$ $π_{0.5}$ : a Vision-Language-Action Model with Open-World Generalization Physical Intelligence Kevin Black Noah Brown James Darpinian Karan Dhabalia ... Homer Walke Anna Walling Haohuan Wang Lili Yu Ury Zhilinsky LM&Ro VLM 109 36 0 22 Apr 2025
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots Nvidia Johan Bjorck Fernando Castañeda Nikita Cherniadev Xingye Da ... Ao Zhang Hao Zhang Yizhou Zhao Ruijie Zheng Yuke Zhu VLM 148 52 0 18 Mar 2025
HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model Jiaming Liu Hao Chen Pengju An Zhuoyang Liu Renrui Zhang ... Chengkai Hou Mengdi Zhao KC alex Zhou Pheng-Ann Heng Shanghang Zhang 143 16 0 13 Mar 2025
OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction Huang Huang Fangchen Liu Letian Fu Tingfan Wu Mustafa Mukadam Jitendra Malik Ken Goldberg Pieter Abbeel LM&Ro VLM 149 10 0 05 Mar 2025
DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control Junjie Wen Yinlin Zhu Jinming Li Zhibin Tang Yaxin Peng Feifei Feng VLM 108 22 0 09 Feb 2025
FAST: Efficient Action Tokenization for Vision-Language-Action Models Karl Pertsch Kyle Stachowicz Brian Ichter Danny Driess Suraj Nair Q. Vuong Oier Mees Chelsea Finn Sergey Levine 132 57 0 17 Jan 2025
Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learning Moritz Reuss Jyothish Pari Pulkit Agrawal Rudolf Lioutikov DiffM MoE 128 8 0 17 Dec 2024
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons Andrew Szot Bogdan Mazoure Omar Attia Aleksei Timofeev Harsh Agrawal Devon Hjelm Zhe Gan Z. Kira Alexander Toshev 3DGS 84 7 0 11 Dec 2024
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation Qixiu Li Yaobo Liang Zeyu Wang Lin Luo Xi Chen ... Jianmin Bao Dong Chen Yuanchun Shi Jiaolong Yang B. Guo LM&Ro 136 38 0 29 Nov 2024
ALOHA Unleashed: A Simple Recipe for Robot Dexterity Tony Zhao Jonathan Tompson Danny Driess Pete Florence Kamyar Ghasemipour Chelsea Finn Ayzaan Wahid 93 76 0 17 Oct 2024
RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation Songming Liu Lingxuan Wu Bangguo Li Hengkai Tan Huayu Chen Zhengyi Wang Ke Xu Hang Su Jun Zhu 119 113 0 10 Oct 2024
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models Matt Deitke Christopher Clark Sangho Lee Rohun Tripathi Yue Yang ... Noah A. Smith Hannaneh Hajishirzi Ross Girshick Ali Farhadi Aniruddha Kembhavi OSLM VLM 77 14 0 25 Sep 2024
TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation Junjie Wen Yinlin Zhu Jinming Li Minjie Zhu Kun Wu ... Ran Cheng Yaxin Peng Chaomin Shen Feifei Feng Jian Tang LM&Ro 114 62 0 19 Sep 2024
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model Chunting Zhou Lili Yu Arun Babu Kushal Tirumala Michihiro Yasunaga Leonid Shamis Jacob Kahn Xuezhe Ma Luke Zettlemoyer Omer Levy DiffM 96 178 0 20 Aug 2024
Robotic Control via Embodied Chain-of-Thought Reasoning Michał Zawalski William Chen Karl Pertsch Oier Mees Chelsea Finn Sergey Levine LRM LM&Ro 120 76 0 11 Jul 2024
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs Shengbang Tong Ellis L Brown Penghao Wu Sanghyun Woo Manoj Middepogu ... Xichen Pan Austin Wang Rob Fergus Yann LeCun Saining Xie 3DV MLLM 111 350 0 24 Jun 2024
OpenVLA: An Open-Source Vision-Language-Action Model Moo Jin Kim Karl Pertsch Siddharth Karamcheti Ted Xiao Ashwin Balakrishna ... Russ Tedrake Dorsa Sadigh Sergey Levine Percy Liang Chelsea Finn LM&Ro VLM 230 477 0 13 Jun 2024
BAKU: An Efficient Transformer for Multi-Task Policy Learning Siddhant Haldar Zhuoran Peng Lerrel Pinto OffRL 88 39 0 11 Jun 2024
Octo: An Open-Source Generalist Robot Policy Octo Model Team Dibya Ghosh Homer Walke Karl Pertsch Kevin Black ... Quan Vuong Ted Xiao Dorsa Sadigh Chelsea Finn Sergey Levine 184 424 0 20 May 2024
Chameleon: Mixed-Modal Early-Fusion Foundation Models Chameleon Team MLLM 182 309 0 16 May 2024
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset Alexander Khazatsky Karl Pertsch Suraj Nair Ashwin Balakrishna Sudeep Dasari ... Thomas Kollar Sergey Levine Chelsea Finn Sergey Levine Chelsea Finn 216 213 0 19 Mar 2024
3D-VLA: A 3D Vision-Language-Action Generative World Model Haoyu Zhen Xiaowen Qiu Peihao Chen Jincheng Yang Xin Yan Yilun Du Yining Hong Chuang Gan LM&Ro VGen PINN 80 100 0 14 Mar 2024
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis Patrick Esser Sumith Kulal A. Blattmann Rahim Entezari Jonas Muller ... Zion English Kyle Lacey Alex Goodwin Yannik Marek Robin Rombach DiffM 284 1,310 0 05 Mar 2024
On Bringing Robots Home Nur Muhammad (Mahi) Shafiullah Anant Rai Haritheja Etukuru Yiqian Liu Ishan Misra Soumith Chintala Lerrel Pinto 113 83 0 27 Nov 2023
CapsFusion: Rethinking Image-Text Data at Scale Qiying Yu Quan-Sen Sun Xiaosong Zhang Yufeng Cui Fan Zhang Yue Cao Xinlong Wang Jingjing Liu VLM 62 60 0 31 Oct 2023
RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking Homanga Bharadhwaj Jay Vakil Mohit Sharma Abhi Gupta Shubham Tulsiani Vikash Kumar LM&Ro 92 128 0 05 Sep 2023
BridgeData V2: A Dataset for Robot Learning at Scale Homer Walke Kevin Black Abraham Lee Moo Jin Kim Maximilian Du ... Andre Wang He Vivek Myers Kuan Fang Chelsea Finn Sergey Levine 81 231 0 24 Aug 2023
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control Anthony Brohan Noah Brown Justice Carbajal Yevgen Chebotar Xi Chen ... Ted Xiao Peng Xu Sichun Xu Tianhe Yu Brianna Zitkovich LM&Ro LRM 148 1,244 0 28 Jul 2023
RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot Haoshu Fang Hongjie Fang Zhenyu Tang Jirong Liu Chenxi Wang Junbo Wang Haoyi Zhu Cewu Lu 105 73 0 02 Jul 2023
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents Hugo Laurenccon Lucile Saulnier Léo Tronchon Stas Bekman Amanpreet Singh ... Siddharth Karamcheti Alexander M. Rush Douwe Kiela Matthieu Cord Victor Sanh 114 242 0 21 Jun 2023
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning Bo Liu Yifeng Zhu Chongkai Gao Yihao Feng Qian Liu Yuke Zhu Peter Stone CLL 144 143 0 05 Jun 2023
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware Tony Zhao Vikash Kumar Sergey Levine Chelsea Finn 92 617 0 23 Apr 2023
Visual Instruction Tuning Haotian Liu Chunyuan Li Qingyang Wu Yong Jae Lee SyDa VLM MLLM 529 4,740 0 17 Apr 2023
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion Cheng Chi Zhenjia Xu S. Feng Eric A. Cousineau Yilun Du Benjamin Burchfiel Russ Tedrake Shuran Song 347 1,189 0 07 Mar 2023
PaLM-E: An Embodied Multimodal Language Model Danny Driess F. Xia Mehdi S. M. Sajjadi Corey Lynch Aakanksha Chowdhery ... Marc Toussaint Klaus Greff Andy Zeng Igor Mordatch Peter R. Florence LM&Ro 109 1,648 0 06 Mar 2023
Flow Matching for Generative Modeling Y. Lipman Ricky T. Q. Chen Heli Ben-Hamu Maximilian Nickel Matt Le OOD 207 1,307 0 06 Oct 2022
Rectified Flow: A Marginal Preserving Approach to Optimal Transport Qiang Liu OT 181 101 0 29 Sep 2022
Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow Xingchao Liu Chengyue Gong Qiang Liu OOD 188 1,003 0 07 Sep 2022
Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr ... Mikolaj Binkowski Ricardo Barreira Oriol Vinyals Andrew Zisserman Karen Simonyan MLLM VLM 382 3,542 0 29 Apr 2022
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets F. Ebert Yanlai Yang Karl Schmeckpeper Bernadette Bucher G. Georgakis Kostas Daniilidis Chelsea Finn Sergey Levine 227 230 0 27 Sep 2021
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning Jun Chen Han Guo Kai Yi Boyang Albert Li Mohamed Elhoseiny VLM 91 227 0 20 Feb 2021
RoboNet: Large-Scale Multi-Robot Learning Sudeep Dasari F. Ebert Stephen Tian Suraj Nair Bernadette Bucher Karl Schmeckpeper Siddharth Singh Sergey Levine Chelsea Finn LM&Ro 81 302 0 24 Oct 2019
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 701 131,652 0 12 Jun 2017
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering Yash Goyal Tejas Khot D. Summers-Stay Dhruv Batra Devi Parikh CoGe 342 3,246 0 02 Dec 2016
Microsoft COCO Captions: Data Collection and Evaluation Server Xinlei Chen Hao Fang Nayeon Lee Ramakrishna Vedantam Saurabh Gupta Piotr Dollar C. L. Zitnick 215 2,478 0 01 Apr 2015