ObjectNet 用于对象识别的大型真实世界的数据集

Ready to help develop the next generation of object recognition algorithms that have robustness, bias, and safety in mind. Controls can remove bias from other datasets machine learning, not just vision. ObjectNet is a large real-world test set for object recognition with control where object backgrounds, rotations, and imaging viewpoints are random. Most scientific experiments have controls, confounds which are removed from the data, to ensure that subjects cannot perform a task by exploiting trivial correlations in the data. Historically, large machine learning and computer vision datasets have lacked such controls. This has resulted in models that must be fine-tuned for new datasets and perform better on datasets than in real-world applications. When tested on ObjectNet, object detectors show a 40-45% drop in performance, with respect to their performance on other benchmarks, due to the controls for biases. Controls make ObjectNet robust to fine-tuning showing only small performance increases. We develop a highly automated platform that enables gathering datasets with controls by crowdsourcing image capturing and annotation. ObjectNet is the same size as the ImageNet test set (50,000 images), and by design does not come paired with a training set in order to encourage generalization. The dataset is both easier than ImageNet – objects are largely centered and unoccluded – and harder, due to the controls. Although we focus on object recognition here, data with controls can be gathered at scale using automated tools throughout machine learning to generate datasets that exercise models in new ways thus providing valuable feedback to researchers. This work opens up new avenues for research in generalizable, robust, and more human-like computer vision and in creating datasets where results are predictive of real-world performance. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems 32, pages 9448–9458. 2019.

本数据集旨在助力开发兼顾鲁棒性、偏置校正与安全性的新一代目标识别算法。其配套控制机制可消除其他数据集（不限于计算机视觉领域）所训练的机器学习模型中的偏置问题。
ObjectNet是一款具备可控性的大型真实世界目标识别测试集，其中目标的背景、旋转角度与成像视角均为随机生成。多数科学实验均会设置控制变量，剔除数据中的混淆因素，确保模型无法利用数据中的琐碎关联完成任务。长期以来，大型机器学习与计算机视觉数据集均缺乏此类控制机制，导致所训练的模型需针对新数据集进行微调，且在基准数据集上的表现优于实际应用场景中的表现。当在ObjectNet上进行测试时，目标检测器的性能相较于其在其他基准数据集上的表现会下降40%至45%，这正是由偏置控制机制所致。该控制机制使得ObjectNet对微调操作具备鲁棒性，微调后仅能带来小幅性能提升。
本研究开发了一套高度自动化的平台，可通过众包图像采集与标注流程，规模化构建具备控制机制的数据集。ObjectNet的规模与ImageNet测试集（50000张图像）一致，且设计上未配套训练集，以推动模型泛化能力的研究。该数据集相较于ImageNet既更简单——目标大多居中且无遮挡——又更具挑战性，这源于其内置的控制机制。
尽管本研究聚焦于目标识别领域，但借助自动化工具，机器学习全领域均可规模化构建具备控制机制的数据集，通过全新方式对模型进行测试，从而为研究者提供极具价值的反馈。本研究为开发具备泛化能力、鲁棒性且更贴近人类视觉的计算机视觉系统，以及构建可预测实际应用性能的数据集开辟了全新研究路径。
Andrei Barbu、David Mayo、Julian Alverio、William Luo、Christopher Wang、Dan Gutfreund、Josh Tenenbaum与Boris Katz。《ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models》，收录于《神经信息处理系统进展（Advances in Neural Information Processing Systems）》第32卷，第9448-9458页，2019年。

搜集汇总

背景与挑战

背景概述

ObjectNet是一个专为对象识别设计的大型真实世界数据集，仅包含测试集，旨在评估视觉系统的泛化能力。数据集包含50,000张图像，控制旋转、背景和视角等变量，包含313个对象类别（113个与ImageNet重叠），通过控制偏差提高模型鲁棒性。