PhraseCut

我们考虑在给定自然语言短语的情况下分割图像区域的问题，并在一个包含 77,262 个图像和 345,486 个短语-区域对的新数据集上对其进行研究。我们的数据集是在 Visual Genome 数据集之上收集的，并使用现有的注释来生成一组具有挑战性的引用短语，其中相应的区域是手动注释的。我们数据集中的短语对应于多个区域，并描述了大量的对象和东西类别以及它们的属性，例如颜色、形状、部件以及与图像中其他实体的关系。我们的实验表明，我们数据集中概念的规模和多样性对现有的最新技术提出了重大挑战。我们系统地处理了这些概念的长尾特性，并提出了一种模块化方法来组合类别、属性和关系线索，其性能优于现有方法。

We consider the problem of segmenting image regions conditioned on natural language phrases, and investigate this task on a novel dataset consisting of 77,262 images and 345,486 phrase-region pairs. Our dataset is curated based on the Visual Genome dataset, where we leverage existing annotations to generate a set of challenging referring phrases, with their corresponding regions manually annotated. The phrases in our dataset correspond to multiple regions, and cover a broad spectrum of object and stuff categories, along with their associated attributes including color, shape, component parts, and relationships with other entities within the image. Our experimental results demonstrate that the scale and diversity of concepts in our dataset pose substantial challenges to prevailing state-of-the-art methods. We systematically address the long-tail distribution property of these concepts, and propose a modular framework that integrates category, attribute, and relationship cues, which outperforms existing baseline approaches.

搜集汇总

背景与挑战

背景概述

PhraseCut是一个基于Visual Genome构建的数据集，包含77,262张图像和345,486个短语-区域对，用于语言引导的图像分割任务。该数据集涵盖了多种对象、属性及其关系，对现有技术提出了挑战，由Adobe Research和马萨诸塞大学阿默斯特分校于2020年发布。