Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

24 October 2018

Silvio Savarese

Li Fei-Fei

Papers citing "Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks"

50 / 93 papers shown

Title
Force-Driven Validation for Collaborative Robotics in Automated Avionics Testing Pietro Dardano Paolo Rocco David Frisini 9 0 0 15 May 2025
BQSched: A Non-intrusive Scheduler for Batch Concurrent Queries via Reinforcement Learning Chenhao Xu Chunyu Chen Jinglin Peng Jiannan Wang Jun Gao OffRL AI4TS 53 0 0 27 Apr 2025
ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface Fangchen Liu Chuanyu Li Yihua Qin Ankit Shaw Jinfeng Xu Pieter Abbeel Rui Chen 56 2 0 08 Apr 2025
HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model Jiaming Liu Hao Chen Pengju An Zhuoyang Liu Renrui Zhang ... Chengkai Hou Mengdi Zhao KC alex Zhou Pheng-Ann Heng Shanghang Zhang 77 9 0 13 Mar 2025
Caging in Time: A Framework for Robust Object Manipulation under Uncertainties and Limited Robot Perception Gaotian Wang Kejia Ren A. S. Morgan Kaiyu Hang 59 1 0 21 Oct 2024
RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation Songming Liu Lingxuan Wu Bangguo Li Hengkai Tan Huayu Chen Zhengyi Wang Ke Xu Hang Su Jun Zhu 45 82 0 10 Oct 2024
iVideoGPT: Interactive VideoGPTs are Scalable World Models Jialong Wu Shaofeng Yin Ningya Feng Xu He Dong Li Haifeng Zhang Mingsheng Long VGen 49 26 0 24 May 2024
Generalize by Touching: Tactile Ensemble Skill Transfer for Robotic Furniture Assembly Hao-ming Lin Radu Corcodel Ding Zhao 40 7 0 26 Apr 2024
Cross-domain Multi-modal Few-shot Object Detection via Rich Text Zeyu Shangguan Daniel Seita Mohammad Rostami ObjD 57 1 0 24 Mar 2024
Symmetry-aware Reinforcement Learning for Robotic Assembly under Partial Observability with a Soft Wrist Hai Nguyen Tadashi Kozuno C. C. Beltran-Hernandez Masashi Hamaya 48 6 0 28 Feb 2024
Integrating DeepRL with Robust Low-Level Control in Robotic Manipulators for Non-Repetitive Reaching Tasks Mehdi Heydari Shahna Seyed Adel Alizadeh Kolagar Jouni Mattila 24 4 0 04 Feb 2024
QUAR-VLA: Vision-Language-Action Model for Quadruped Robots Pengxiang Ding Han Zhao Wenxuan Song Zhitao Wang Zhenyu Wei Shangke Lyu Ningxi Yang Donglin Wang 34 19 0 22 Dec 2023
Intrinsic Language-Guided Exploration for Complex Long-Horizon Robotic Manipulation Tasks Wenke Huang Filippos Christianos Zhibin Li 40 8 0 28 Sep 2023
Few-Shot Learning of Force-Based Motions From Demonstration Through Pre-training of Haptic Representation Marina Y. Aoyama João Moura Namiko Saito S. Vijayakumar 24 5 0 08 Sep 2023
PoseFusion: Robust Object-in-Hand Pose Estimation with SelectLSTM Yuyang Tu Junnan Jiang Shuang Li Norman Hendrich Miaopeng Li Jianwei Zhang 29 7 0 10 Apr 2023
Efficient Skill Acquisition for Complex Manipulation Tasks in Obstructed Environments Jun Yamada J. Collins Ingmar Posner 36 8 0 06 Mar 2023
Zero-Shot Transfer of Haptics-Based Object Insertion Policies Samarth Brahmbhatt A. Deka Andrew Spielberg M. Muller 9 6 0 29 Jan 2023
See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation Hao Li Yizhi Zhang Junzhe Zhu Shaoxiong Wang Michelle A. Lee Huazhe Xu Edward H. Adelson Li Fei-Fei Ruohan Gao Jiajun Wu 32 60 0 07 Dec 2022
Prim-LAfD: A Framework to Learn and Adapt Primitive-Based Skills from Demonstrations for Insertion Tasks Zheng Wu Wenzhao Lian Changhao Wang Mengxi Li S. Schaal Masayoshi Tomizuka 8 10 0 02 Dec 2022
Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding Priya Sundaresan Suneel Belkhale Dorsa Sadigh 43 33 0 26 Nov 2022
SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Rendering Jun Lv Yunhai Feng Cheng Zhang Shu Zhao Lin Shao Cewu Lu 18 24 0 27 Oct 2022
CFVS: Coarse-to-Fine Visual Servoing for 6-DoF Object-Agnostic Peg-In-Hole Assembly B. Lu Tung-I Chen Hsin-Ying Lee Winston H. Hsu 38 10 0 19 Sep 2022
Sim-to-Real Transfer of Robotic Assembly with Visual Inputs Using CycleGAN and Force Control C. Yuan Yunlei Shi Qian Feng Chunyang Chang Zhaopeng Chen Alois C. Knoll Jianwei Zhang 29 8 0 30 Aug 2022
Semi-Supervised Disentanglement of Tactile Contact~Geometry from Sliding-Induced Shear A. Gupta Alex Church Nathan Lepora 26 2 0 26 Aug 2022
Maximizing the Use of Environmental Constraints: A Pushing-Based Hybrid Position/Force Assembly Skill for Contact-Rich Tasks Yunlei Shi Zhaopeng Chen Lin Cong Yansong Wu Martin Craiu-Müller C. Yuan Chunyang Chang Lei Zhang Jianwei Zhang 17 0 0 12 Aug 2022
Impact Makes a Sound and Sound Makes an Impact: Sound Guides Representations and Explorations Xufeng Zhao C. Weber Muhammad Burhan Hafez S. Wermter 28 8 0 04 Aug 2022
Learning Haptic-based Object Pose Estimation for In-hand Manipulation Control with Underactuated Robotic Hands Osher Azulay Inbar Ben-David A. Sintov 35 14 0 06 Jul 2022
Integrating High-Resolution Tactile Sensing into Grasp Stability Prediction Lachlan Chumbley Morris Gu Rhys Newbury Jurgen Leitner Akansel Cosgun 25 2 0 12 Jun 2022
Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning Maximilian Du Olivia Y. Lee Suraj Nair Chelsea Finn OffRL 59 33 0 30 May 2022
Learning Dense Reward with Temporal Variant Self-Supervision Yuning Wu Jieliang Luo Hui Li SSL 18 0 0 20 May 2022
Action Conditioned Tactile Prediction: case study on slip prediction Willow Mandil Kiyanoush Nazari E. AmirGhalamzan 27 16 0 19 May 2022
Learning A Simulation-based Visual Policy for Real-world Peg In Unseen Holes Liangru Xie Hongxiang Yu Kechun Xu Tong Yang Minhang Wang Haojian Lu R. Xiong Yue Wang 31 0 0 09 May 2022
Learning to Fill the Seam by Vision: Sub-millimeter Peg-in-hole on Unseen Shapes in Real World Liang Xie Hongxiang Yu Yinghao Zhao Haodong Zhang Zhongxiang Zhou Minhang Wang Yue Wang R. Xiong 22 9 0 16 Apr 2022
Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness Paul Pu Liang 30 4 0 14 Apr 2022
Visuo-Haptic Object Perception for Robots: An Overview Nicolás Navarro-Guerrero Sibel Toprak Josip Josifovski L. Jamone 30 36 0 22 Mar 2022
Multiscale Sensor Fusion and Continuous Control with Neural CDEs Sumeet Singh Francis McCann Ramirez Jacob Varley Andy Zeng Vikas Sindhwani 49 3 0 16 Mar 2022
Multi-View Dreaming: Multi-View World Model with Contrastive Learning Akira Kinose Masashi Okada Ryogo Okumura T. Taniguchi OffRL 21 10 0 15 Mar 2022
MIRROR: Differentiable Deep Social Projection for Assistive Human-Robot Communication Kaiqi Chen J. Fong Harold Soh 24 10 0 06 Mar 2022
COMPASS: Contrastive Multimodal Pretraining for Autonomous Systems Shuang Ma Sai H. Vemprala Wenshan Wang Jayesh K. Gupta Yale Song Daniel J. McDuff Ashish Kapoor SSL 37 9 0 20 Feb 2022
Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation Rishabh Jangir Nicklas Hansen Sambaran Ghosal Mohit Jain Xiaolong Wang 32 66 0 19 Jan 2022
Contact-Rich Manipulation of a Flexible Object based on Deep Predictive Learning using Vision and Tactility Hideyuki Ichiwara Hiroshi Ito Kenjiro Yamamoto Hiroki Mori Tetsuya Ogata 21 21 0 13 Dec 2021
Balancing Efficiency and Comfort in Robot-Assisted Bite Transfer Suneel Belkhale E. Gordon Yuxiao Chen S. Srinivasa T. Bhattacharjee Dorsa Sadigh 47 19 0 22 Nov 2021
Learning Ultrasound Scanning Skills from Human Demonstrations Xutian Deng Ziwei Lei Yi Wang Miao Li 17 15 0 09 Nov 2021
Learning Robotic Ultrasound Scanning Skills via Human Demonstrations and Guided Explorations Xutian Deng Yiting Chen Fei Chen Miao Li 13 23 0 02 Nov 2021
SEED: Series Elastic End Effectors in 6D for Visuotactile Tool Use H.J. Terry Suh Naveen Kuppuswamy Tao Pang P. Mitiguy A. Alspach Russ Tedrake 24 12 0 02 Nov 2021
Learning Robotic Manipulation Skills Using an Adaptive Force-Impedance Action Space Maximilian Ulmer Elie Aljalbout Sascha Schwarz Sami Haddadin 18 6 0 19 Oct 2021
Seeking Visual Discomfort: Curiosity-driven Representations for Reinforcement Learning Elie Aljalbout Maximilian Ulmer Rudolph Triebel 24 2 0 02 Oct 2021
Making Curiosity Explicit in Vision-based RL Elie Aljalbout Maximilian Ulmer Rudolph Triebel OffRL 34 2 0 28 Sep 2021
ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations Ruohan Gao Yen-Yu Chang Shivani Mall Li Fei-Fei Jiajun Wu 41 79 0 16 Sep 2021
Exploring Data Aggregation and Transformations to Generalize across Visual Domains Antono DÍnnocente OOD 33 0 0 20 Aug 2021