Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

28 July 2019

Silvio Savarese

Papers citing "Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks"

33 / 33 papers shown

Title
VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation Chaofan Zhang Peng Hao Xiaoge Cao Xiaoshuai Hao Shaowei Cui Shuo Wang 32 0 0 14 May 2025
ViTa-Zero: Zero-shot Visuotactile Object 6D Pose Estimation Hongyu Li James Akl Srinath Sridhar Tye Brady Taskin Padir 37 0 0 17 Apr 2025
Force-Aware Autonomous Robotic Surgery Alaa Eldin Abdelaal Jiaying Fang Tim N. Reinhart Jacob A. Mejia Tony Zhao Jeannette Bohg Allison M. Okamura OOD 40 0 0 20 Jan 2025
Robust Contact-rich Manipulation through Implicit Motor Adaptation Teng Xue Amirreza Razmjoo Suhan Shetty Sylvain Calinon 102 1 0 16 Dec 2024
What to align in multimodal contrastive learning? Benoit Dufumier J. Castillo-Navarro D. Tuia Jean-Philippe Thiran 29 3 0 11 Sep 2024
FORGE: Force-Guided Exploration for Robust Contact-Rich Manipulation under Uncertainty Michael Noseworthy Bingjie Tang Bowen Wen Ankur Handa Nicholas Roy Nicholas Roy Dieter Fox Yashraj S. Narang Iretiayo Akinola Iretiayo Akinola 49 8 0 08 Aug 2024
Multi-modal perception for soft robotic interactions using generative models Enrico Donato Egidio Falotico T. G. Thuruthel 26 2 0 05 Apr 2024
PseudoTouch: Efficiently Imaging the Surface Feel of Objects for Robotic Manipulation Adrian Rofer Nick Heppert Abdallah Ayman Eugenio Chisari Abhinav Valada 29 1 0 22 Mar 2024
BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning Jianbo Zhao Hao Chen Qingyao Tian Jian Chen Bingyu Yang Hongbin Liu 37 1 0 03 Mar 2024
On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer Elie Aljalbout Felix Frank Maximilian Karl Patrick van der Smagt 14 20 0 06 Dec 2023
RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot Haoshu Fang Hongjie Fang Zhenyu Tang Jirong Liu Chenxi Wang Junbo Wang Haoyi Zhu Cewu Lu 38 58 0 02 Jul 2023
Bayesian and Neural Inference on LSTM-based Object Recognition from Tactile and Kinesthetic Information Francisco Pastor Jorge García-González J. Gandarias D. Medina Pau Closas A. García-Cerezo J. M. G. D. Gabriel BDL 14 41 0 10 Jun 2023
Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play Irmak Güzey Ben Evans Soumith Chintala Lerrel Pinto 64 65 0 21 Mar 2023
Recent Advances of Deep Robotic Affordance Learning: A Reinforcement Learning Perspective Xintong Yang Ze Ji Jing Wu Yunyu Lai 33 12 0 09 Mar 2023
DSL-Assembly: A Robust and Safe Assembly Strategy Yi Liu OffRL 14 0 0 21 Feb 2023
Zero-Shot Transfer of Haptics-Based Object Insertion Policies Samarth Brahmbhatt A. Deka Andrew Spielberg M. Muller 9 5 0 29 Jan 2023
See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation Hao Li Yizhi Zhang Junzhe Zhu Shaoxiong Wang Michelle A. Lee Huazhe Xu Edward H. Adelson Li Fei-Fei Ruohan Gao Jiajun Wu 27 58 0 07 Dec 2022
Automatic Evaluation of Excavator Operators using Learned Reward Functions Pranav Agarwal M. Teichmann Sheldon Andrews Samira Ebrahimi Kahou OffRL 27 2 0 15 Nov 2022
Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations Yilun Hao Ruinan Wang Zhangjie Cao Zihan Wang Yuchen Cui Dorsa Sadigh 29 2 0 16 Sep 2022
Factory: Fast Contact for Robotic Assembly Yashraj S. Narang Kier Storey Iretiayo Akinola Miles Macklin Philipp Reist ... Ádám Moravánszky Gavriel State Michelle Lu Ankur Handa D. Fox 30 74 0 07 May 2022
Learning Goal-Oriented Non-Prehensile Pushing in Cluttered Scenes Nils Dengler D. Grossklaus Maren Bennewitz 11 16 0 04 Mar 2022
Multi-Modal Fusion in Contact-Rich Precise Tasks via Hierarchical Policy Learning Piaopiao Jin Yi-Bin Lin Yanchao Tan Tiefeng Li Wei Yang 16 1 0 17 Feb 2022
End-to-end Reinforcement Learning of Robotic Manipulation with Robust Keypoints Representation Tianying Wang En Yen Puang Marcus Lee Y. Wu Wei Jing SSL 23 5 0 12 Feb 2022
Large-Dimensional Multibody Dynamics Simulation Using Contact Nodalization and Diagonalization Jeongmin Lee Minji Lee Dongjun Lee AI4CE 20 4 0 23 Jan 2022
Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Models Iman Nematollahi Erick Rosete-Beas Adrian Rofer Tim Welschehold Abhinav Valada Wolfram Burgard 19 15 0 25 Nov 2021
Active Extrinsic Contact Sensing: Application to General Peg-in-Hole Insertion Sangwoon Kim Alberto Rodriguez 44 64 0 07 Oct 2021
Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation Yifeng Zhu Peter Stone Yuke Zhu 45 60 0 28 Sep 2021
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning Paul Pu Liang Yiwei Lyu Xiang Fan Zetian Wu Yun Cheng ... Peter Wu Michelle A. Lee Yuke Zhu Ruslan Salakhutdinov Louis-Philippe Morency VLM 29 159 0 15 Jul 2021
Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers Ruihan Yang Minghao Zhang Nicklas Hansen Huazhe Xu Xiaolong Wang OffRL 13 101 0 08 Jul 2021
Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning Kaiqi Chen Yong Lee Harold Soh SSL 25 20 0 06 Jul 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 50 973 0 04 Mar 2021
Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors Michelle A. Lee Matthew Tan Yuke Zhu Jeannette Bohg 41 25 0 01 Dec 2020
Towards Learning Controllable Representations of Physical Systems Kevin Haninger R. Vicente-Garcia J. Krüger 23 1 0 16 Nov 2020