GradDA – A novel dataset for investigating domain shifts in image classification

A domain shift occurs when the testing data is drawn from a distribution different from that of the training dataset. This shift presents a significant challenge and may compromise the performance of machine learning models, which leads to poor generalization. Over the past years, various models have been developed and evaluated on benchmark datasets such as VisDA, Office-Home and DomainNet. These datasets consist of discrete domains with different object classes. However, a notable limitation when addressing the domain shift is the absence of data samples where the exact same object exists in both domains. We propose a new dataset designed to address this challenge. In particular, we introduce a domain shift from a purely synthetic style (grey object on white background) to a more realistic appearance (object with texture against a realistic background) with differential modifications, which enables the representation of the same object in both synthetic and real domains, consequently facilitating the analysis of a transition between the two domains. The dataset comprises five distinct classes (Airplane, Bicycle, Bus, Car, Train), with multiple objects per class. Additionally, each object is depicted from 20 different perspectives, resulting in a total of 101 images per perspective that captures the transition from pure synthetic to a more real-world-like domain. This dataset offers a unique opportunity to investigate the impact of domain shift on model performance in classification tasks, as it focuses solely on domain changes without other interfering effects. It is the objective of our work to trigger new discussions about the domain shift problem, and how it can be tackled with alternative data driven model designs.

当测试数据的分布与训练数据集的分布不一致时，便会产生域偏移（domain shift）问题。此类偏移会给机器学习模型带来显著挑战，甚至可能降低模型性能，导致泛化能力不佳。
近年来，学界已开发出多种机器学习模型，并在VisDA、Office-Home与DomainNet等基准数据集（benchmark datasets）上开展了评估。此类数据集由包含不同目标类别的离散域构成。然而，当前针对域偏移问题的研究存在一项显著局限：两类域中不存在完全相同的目标样本。
为此，我们提出一款全新的数据集以应对该挑战。具体而言，我们构建了从纯合成风格（白色背景下的灰色目标）到更逼真外观（带有纹理的目标搭配真实背景）的域偏移，并通过差异化修改实现了同一目标在合成域与真实域中的统一表征，从而便于分析两类域之间的过渡过程。
该数据集包含5个不同类别：飞机、自行车、公交车、汽车与火车，每个类别下包含多个目标。此外，每个目标均从20个不同视角进行采集，且每个视角下包含101张图像，可完整呈现从纯合成域到类真实世界域的过渡过程。
本数据集为探究域偏移对分类任务中模型性能的影响提供了独特的研究契机，因其仅聚焦域分布变化，未引入其他干扰因素。
本研究旨在推动学界围绕域偏移问题展开更多讨论，并探索如何通过替代的数据驱动模型设计来解决该问题。