Dexterous object manipulation by a multi-fingered robotic hand with visual-tactile fingertip sensors

In this paper, a novel visual-tactile sensor is proposed; additionally, an object manipulation method for a multi-fingered robotic hand grasping an object is proposed by detecting a contact position using the visual-tactile sensor. The visual-tactile sensor is composed of a hemispheric fingertip made of soft silicone with a hollow interior and a general USB camera located inside the fingertip to detect the displacement of the many point markers embedded in the silicone. The deformation of each point marker due to a contact force is measured, and a contact position is estimated reliably through a novel method of creating virtual points to determine the point clouds. The aim is to demonstrate both the estimation performance of the new visual-tactile sensor and its usefulness in a grasping and manipulation task. By using the contact position obtained from the proposed sensor and the position of each fingertip obtained from kinematics, the position and orientation of a grasped object are estimated and controlled. The effectiveness of the method is illustrated through numerical simulation and its practical use is demonstrated through grasping and manipulating experiments.