Robust Visual Servoing under Human Supervision for Assembly Tasks

We propose a framework enabling mobile manipulators to reliably complete pick-and-place tasks for assembling structures from construction blocks. The picking uses an eye-in-hand visual servoing controller for object tracking with Control Barrier Functions (CBFs) to ensure fiducial markers in the blocks remain visible. An additional robot with an eye-to-hand setup ensures precise placement, critical for structural stability. We integrate human-in-the-loop capabilities for flexibility and fault correction and analyze robustness to camera pose errors, proposing adapted barrier functions to handle them. Lastly, experiments validate the framework on 6-DoF mobile arms.

@article{fernandez-ayala2025_2504.12506,
  title={ Robust Visual Servoing under Human Supervision for Assembly Tasks },
  author={ Victor Nan Fernandez-Ayala and Jorge Silva and Meng Guo and Dimos V. Dimarogonas },
  journal={arXiv preprint arXiv:2504.12506},
  year={ 2025 }
}