Vident-real: an intra-oral video dataset for multi-task learning

We introduce Vident-real, a large dataset of 100 video sequences of intra-oral scenes from real conservative dental treatments performed at the Medical University of Gdańsk, Poland. The dataset can be used for multi-task learning methods including:

video enhancement
video segmentation
motion estimation
video stabilization

The dataset allows for training and validating models on multiple vision-based tasks in challenging real conditions characterized by compromised visibility. The recordings were acquired with a tiny micro-camera firmly attached to dental handpieces with various dental burs and tools. The dental scenes were crowded due to the presence of dental tools and artifacts and featured occlusions, appearance variations, tool-teeth interactions, bleeding, motion blur, light reflections, splashing water and other fluids, and camera fouling.
Since the sequences recorded real dental treatment procedures, collecting target labels from referential, additional sensors in confined spaces is impractical. In the whole dataset, each input video frame, which is corrupted due to sensor miniaturization and other common adversarial factors, is paired with pseudo-labels of:

enhanced frame
segmented teeth
teeth-based homography (4 dof / similarity) between consecutive frames

Vident-real contains 100 real intra-oral videos of 70K frames recorded during conservative treatment procedures. All sequences were recorded in RAW 10-bit format through a wide-angle lens with the sensor's resolution of 800x800 pixels and high sampling frequency ranging from 55 to 60 Hz. The RAW images were debayerized and stored in JPEG format. Sensor's gain and integration time were manually adjusted to each patient's intra-oral cavity to account for on-site low-light conditions thereby improving visibility and colors in the dynamically changing environment.
A miniaturized camera affixed to a dental handpiece could allow dentists to continuously monitor the progress of conservative dental interventions. Camera-augmented dental interventions hold the potential to facilitate dental training and education, optimize workflow ergonomics, and improve patient outcomes. For safe and effective navigation in the mouth, the necessary miniaturization of sensors and optics introduces artifacts to video streams. The inevitable camera shakes result in eye fatigue. The unique challenges posed by intra-oral conditions, such as noise, blur, texture paucity, light variations, shadows, reflections, and fluid dynamics make continuous macro-visualization of complex dental scenes on customized displays difficult. Enhancement of videos acquired in these challenging conditions appears as a natural step towards advancing the field of Video-Assisted Dentistry (VAD), enabling clearer view of the teeth, fractures, gums, blood, cavities, fillings, dentine, pulp, and dental tools.

我们提出Vident-real数据集，该数据集包含100段来自波兰格但斯克医科大学开展的真实保守牙科治疗场景的口腔内视频序列。本数据集可用于多任务学习方法，涵盖：视频增强、视频分割、运动估计、视频稳像。
该数据集支持在能见度受限的真实复杂场景下，对多视觉任务相关模型进行训练与验证。本次录制采用一款微型相机，该相机牢固安装在搭载各类牙科车针与工具的牙科手机上。牙科场景中因存在牙科工具与成像伪影而显得拥挤，且存在遮挡、外观变化、工具与牙齿交互、出血、运动模糊、光线反射、飞溅的水与其他体液以及相机污染等多种挑战。
由于该数据集的序列均为真实牙科治疗过程的录制，在密闭的口腔空间内通过参考额外传感器获取目标标签并不现实。在全数据集内，每一张因传感器微型化及其他常见对抗性因素而产生失真的输入视频帧，均与以下伪标签配对：增强帧、牙齿分割掩码、连续帧间基于牙齿的单应性（homography，4自由度/相似变换）。
Vident-real数据集包含100段真实口腔内视频，共计7万帧，均采集自保守牙科治疗过程中。所有序列均通过广角镜头以10位RAW格式录制，传感器分辨率为800×800像素，采样频率介于55至60Hz之间。所采集的RAW图像均经过去拜耳化处理，并以JPEG格式存储。针对每位患者的口腔内环境，科研人员手动调整传感器增益与曝光积分时间，以适配现场低光照条件，从而改善动态变化环境中的能见度与色彩表现。
将微型相机固定于牙科手机上，可帮助牙医持续监测保守牙科干预的操作进度。搭载相机的牙科干预手段有望助力牙科培训与教育、优化工作流程的人机工程学表现，并改善患者预后效果。为实现口腔内安全且有效的操作导航，传感器与光学系统必须进行微型化设计，这会给视频流引入成像伪影。不可避免的相机抖动还会引发视觉疲劳。口腔内环境带来的独特挑战，包括噪声、模糊、纹理匮乏、光线变化、阴影、反射以及流体动态等，使得在定制显示器上对复杂牙科场景进行持续宏观可视化变得困难。
对这类复杂场景下采集的视频进行增强，是推动视频辅助牙科（Video-Assisted Dentistry, VAD）领域发展的自然一步，可帮助医护人员更清晰地观察牙齿、牙折、牙龈、血液、龋齿、补牙材料、牙本质、牙髓以及牙科工具。