QLEVR Dataset

Synthetic datasets have successfully been used to probe visual question-answering datasets for their reasoning abilities. CLEVR, for example, tests a range of visual reasoning abilities. The questions in CLEVR focus on comparisons of shapes, colors, and sizes, numerical reasoning, and existence claims. This paper introduces a minimally biased, diagnostic visual question-answering dataset, QLEVR, that goes beyond existential and numerical quantification and focus on more complex quantifiers and their combinations, e.g., asking whether there are more than two red balls that are smaller than at least three blue balls in an image. We describe how the dataset was created and present a first evaluation of state-of-the-art visual question-answering models, showing that QLEVR presents a formidable challenge to our current models.

Description and image from: QLEVR Dataset Generation

合成数据集已被成功应用于探究视觉问答数据集的推理能力。以 CLEVR 为例，它测试了一系列视觉推理能力。CLEVR 中的问题主要集中在形状、颜色和大小的比较，数值推理以及存在性断言等方面。本文提出了一种最小偏差、具有诊断性的视觉问答数据集 QLEVR，该数据集超越了存在性和数值量化，并专注于更复杂的量化及其组合，例如询问在图像中是否有多于两个小于至少三个蓝色球的红球。我们描述了数据集的创建过程，并展示了当前最先进视觉问答模型的首次评估，表明 QLEVR 对我们的当前模型构成了严峻的挑战。