A Multi-View RGB Dataset for Marker-less Needle Tracking

Our dataset contains multi-view RGB video frame sequences for training a markerless needle-tracking system. The data was collected using Logitech StreamCam Webcams (1920×1080 resolution, 60 FPS, 78° field of view). The dataset contains over 37,000 annotated frames spanning a wide range of needle movements, backgrounds, and lighting conditions, taken in sequences ranging from 2 to 30 seconds. All the sequences were recorded at 60 FPS and stored at 800×450 resolution, with synchronized frames from one or two camera perspectives.We captured two different needle types across various scenarios, ranging from standard laboratory settings to CT environments with and without phantoms in our dataset. Further, we included diverse lighting conditions, backgrounds, needle-handling techniques, and viewing angles, as well as different people handling the needle, to ensure sufficient coverage of different needle interactions and appearances. The temporal continuity across the sequences is ensured.For ground-truth mask generation, we used a ChArUco marker shield mounted on needle handles, enabling an automated annotation with minimal manual effort. The segmentation masks are provided as heatmaps for three needle components (tip, base, and shaft), with intensity peaks of 18 that fade from the center of each needle component.