Digital twins enhanced 2D vision-based bin-picking with monocular depth estimation

Bin picking is a fundamental task in industrial automation, where robots are required to identify, locate, and grasp objects from cluttered environments. Accurate pose estimation is crucial in this process, as it directly impacts on the robot’s ability to execute precise and successful grasps. While 3D vision systems offer high precision, their high cost limits their accessibility in budget-constrained environments. To overcome this challenge, this paper proposes a 2D vision-based approach that utilizes RGB cameras for pose estimation in bin-picking applications. The proposed method integrates deep learning techniques, combining Yolov8 Instance Segmentation for object detection and segmentation with MiDaS, a multi-scale network for monocular depth estimation. This enables effective depth estimation without the need for expensive 3D sensors. The system estimates the relative depth between the camera and objects, while the object’s distance from the camera is computed using a metric based on its width and focal length. Furthermore, digital twins are introduced to simulate and optimize the bin-picking system. The digital twins represent the physical robotic environment, allowing for real-time simulations of grasping strategies, collision avoidance, and trajectory planning. By mirroring real-world conditions, it enables precise calibration of grasping parameters, reducing trial-and-error in physical experiments. This integration enhances the system’s efficiency, ensuring accurate and reliable pickup point estimation. Experimental results demonstrate that the proposed approach successfully predicts pickup points with high reliability, making it well-suited for applications where minor positional deviations are acceptable. The combination of 2D vision-based pose estimation and digital twins simulation provides a cost-effective and efficient solution for robotic bin picking, bridging the gap between affordability and precision.