Toward a Robotic Chemist with Multimodal Sense of Touch via Precise Marker Tracking and Vector Analysis

Tactile perception is crucial for safe and effective robotic grasping and manipulation of deformable, fragile objects, as well as intuitive human‒robotic interactions. Visual-based tactile sensors have emerged as one of the most promising solutions to achieve human-like touch perception. Current visual tactile sensors often require large training data and high computational costs for each individual design, limiting the widespread adoptions in practical applications. Here, we report the PixelTouch, a vision-based tactile sensor that provides contact edge, high spatiotemporal resolution pressure mapping (0.3 mm, 63 Hz), and self-decoupled multiaxis force measurement, via micrometre accuracy marker tracking (0.4 µm) and vector analysis. The proposed method is versatile, low computational cost, transformative, can be readily implemented in any vision-based tactile sensor. We demonstrate a PixelTouch equipped robot autonomously completes consecutive challenging tasks in chemical experiments, including handling of a rubber-head pipette, interaction with human collaborator, liquid stirring and viscosity assessment.