The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning

2 November 2023

Pieter Abbeel

Papers citing "The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning"

31 / 31 papers shown

Title
ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface Fangchen Liu Chuanyu Li Yihua Qin Ankit Shaw Jinfeng Xu Pieter Abbeel Rui Chen 125 5 0 08 Apr 2025
Canonical Representation and Force-Based Pretraining of 3D Tactile for Dexterous Visuo-Tactile Policy Learning Tianhao Wu Jinzhou Li Jiyao Zhang Mingdong Wu Hao Dong SSL 94 8 0 26 Sep 2024
Low Fidelity Visuo-Tactile Pretraining Improves Vision-Only Manipulation Performance Selam Gano Abraham George A. Farimani OnRL 83 1 0 21 Jun 2024
See to Touch: Learning Tactile Dexterity through Visual Incentives Irmak Güzey Yinlong Dai Ben Evans Soumith Chintala Lerrel Pinto 83 37 0 21 Sep 2023
General In-Hand Object Rotation with Vision and Touch Haozhi Qi Brent Yi Sudharshan Suresh Mike Lambeta Yi Ma Roberto Calandra Jitendra Malik 111 92 0 18 Sep 2023
Real-World Robot Learning with Masked Visual Pre-training Ilija Radosavovic Tete Xiao Stephen James Pieter Abbeel Jitendra Malik Trevor Darrell SSL 235 254 0 06 Oct 2022
Masked World Models for Visual Control Younggyo Seo Danijar Hafner Hao Liu Fangchen Liu Stephen James Kimin Lee Pieter Abbeel OffRL 168 149 0 28 Jun 2022
Multimodal Masked Autoencoders Learn Transferable Representations Xinyang Geng Hao Liu Lisa Lee Dale Schuurams Sergey Levine Pieter Abbeel 79 118 0 27 May 2022
Visual-Tactile Multimodality for Following Deformable Linear Objects Using Reinforcement Learning Leszek Pecyna Siyuan Dong Shan Luo 53 22 0 31 Mar 2022
Masked Visual Pre-training for Motor Control Tete Xiao Ilija Radosavovic Trevor Darrell Jitendra Malik SSL 113 250 0 11 Mar 2022
Tac3D: A Novel Vision-based Tactile Sensor for Measuring Forces Distribution and Estimating Friction Coefficient Distribution Lunwei Zhang Yue Wang Yao Jiang 43 35 0 13 Feb 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 477 7,827 0 11 Nov 2021
ReSkin: versatile, replaceable, lasting tactile skins Raunaq M. Bhirangi Tess Hellebrekers Carmel Majidi Abhinav Gupta SyDa 90 77 0 29 Oct 2021
Ego4D: Around the World in 3,000 Hours of Egocentric Video Kristen Grauman Andrew Westbury Eugene Byrne Zachary Chavis Antonino Furnari ... Mike Zheng Shou Antonio Torralba Lorenzo Torresani Mingfei Yan Jitendra Malik EgoV 410 1,114 0 13 Oct 2021
Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning Viktor Makoviychuk Lukasz Wawrzyniak Yunrong Guo Michelle Lu Kier Storey ... David Hoeller Nikita Rudin Arthur Allshire Ankur Handa Gavriel State 180 1,091 0 24 Aug 2021
An Empirical Study of Training Self-Supervised Vision Transformers Xinlei Chen Saining Xie Kaiming He ViT 161 1,871 0 05 Apr 2021
SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulation Chen Wang Shaoxiong Wang Branden Romero Filipe Veiga Edward H. Adelson 71 108 0 28 Jan 2021
Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation Thomas Bi Carmelo Sferrazza Raffaello DÁndrea 146 36 0 07 Jan 2021
Sim-to-real for high-resolution optical tactile sensing: From images to 3D contact force distributions Carmelo Sferrazza Raffaello DÁndrea DiffM 61 32 0 21 Dec 2020
Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation Albert Zhan Philip Zhao Lerrel Pinto Pieter Abbeel Michael Laskin SSL DRL 66 21 0 14 Dec 2020
robosuite: A Modular Simulation Framework and Benchmark for Robot Learning Yuke Zhu J. Wong Ajay Mandlekar Roberto Martín-Martín Abhishek Joshi Soroush Nasiriany Yifeng Zhu Soroush Nasiriany Yifeng Zhu 177 445 0 25 Sep 2020
Deep Reinforcement Learning for Tactile Robotics: Learning to Type on a Braille Keyboard Alex Church John Lloyd R. Hadsell Nathan Lepora 74 31 0 06 Aug 2020
Tactile Dexterity: Manipulation Primitives with Tactile Feedback F. Hogan José Ballester Siyuan Dong Alberto Rodriguez 192 103 0 08 Feb 2020
Cable Manipulation with a Tactile-Reactive Gripper Y. She Shaoxiong Wang Siyuan Dong N. Sunil Alberto Rodriguez Edward H. Adelson 103 258 0 03 Oct 2019
Tactile-Based Insertion for Dense Box-Packing Siyuan Dong Alberto Rodriguez 156 55 0 12 Sep 2019
Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning Sandy H. Huang Martina Zambelli Jackie Kay M. Martins Yuval Tassa P. Pilarski R. Hadsell 67 51 0 20 Mar 2019
Learning Latent Space Dynamics for Tactile Servoing Giovanni Sutanto Nathan D. Ratliff Rui Luo Yaodong Yang Yixiao Liu Ankur Handa Dieter Fox 101 31 0 08 Nov 2018
Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks Michelle A. Lee Yuke Zhu K. Srinivasan Parth Shah Silvio Savarese Li Fei-Fei Animesh Garg Jeannette Bohg SSL 93 370 0 24 Oct 2018
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 547 19,296 0 20 Jul 2017
Context Encoders: Feature Learning by Inpainting Deepak Pathak Philipp Krahenbuhl Jeff Donahue Trevor Darrell Alexei A. Efros SSL 69 5,300 0 25 Apr 2016
High-Dimensional Continuous Control Using Generalized Advantage Estimation John Schulman Philipp Moritz Sergey Levine Michael I. Jordan Pieter Abbeel OffRL 133 3,439 0 08 Jun 2015