EyeGuide - From Gaze Data to Instance Segmentation

Obtaining precise instance-level segmentations is a challenging task in machine learning. Especially for objects with complex and non-convex geometries or with partial occlusions scribble, bounding boxes or user clicks are often provided to guide the segmentation. In the paper "EyeGuide - From Gaze Data to Instance Segmentation", we explore the usage of a remote eye tracking system to generate gaze data as an additional input for object segmentation models. The gaze data is recorded during routine image inspections (i.e. without giving a particular task) and is used as an additional input to train neural networks. The training and evaluation was done on the PascalVOC2012 train and a part of the Cellpose dataset. The corresponding raw gaze data are published here.

在机器学习领域，获取精准的实例级分割（instance-level segmentation）是一项极具挑战性的任务。尤其是针对具备复杂非凸几何形状或存在部分遮挡的物体，研究人员通常会借助涂鸦标注（scribble）、边界框（bounding box）或用户点击操作来引导分割任务。在《EyeGuide——从注视数据到实例级分割》（"EyeGuide - From Gaze Data to Instance Segmentation"）一文中，我们探索了利用远程眼动追踪系统生成注视数据（gaze data），并将其作为目标分割模型的额外输入模态的研究路径。该注视数据采集于常规图像巡检过程中（即无特定任务设定的场景），并被用作训练神经网络（neural networks）的额外输入。模型的训练与评估在PascalVOC2012训练集以及部分Cellpose数据集上完成，对应的原始注视数据现已在此发布。