ActivityNet-FOIL

为了测试EMScore对识别包含不准确视觉概念的箔片标题的敏感性，我们遵循foil-COCO数据集来更改ActivityNet-Entities测试数据集，以构建ActivityNet-箔片数据集。在ActivityNet-实体中，每个视频都有两个相应的段落。我们使用两个段落中的一个来构建正确的箔对，并使用另一个作为基于参考的度量的参考。每个段落在不同的时间戳中有大约3个句子，每个句子中的视觉概念都基于注释边界框。通过用类似但错误的概念替换原始视觉概念来创建箔标题。我们的数据生成过程有三个主要步骤: 首先，我们收集所有视觉概念，并过滤掉低频概念。然后，我们将属于同一超级类别的单词配对在一起 (例如river-pool，shirt-shoe，catdog)。最后，我们获得了2,191个正确的箔对，其中每个视觉概念有大约13个箔对。其次，我们将原始正确字幕中的视觉概念替换为配对的箔候选，以构建候选箔字幕。每个正确的标题都有多个候选箔标题。第三，对于每个正确的标题，我们通过选择最低的困惑度候选来挖掘最困难的箔标题。最后，我们创建1900正确的箔片段落对，箔片段落中的至少一个标题包含箔片视觉概念。如图4所示，每个视频包含一个正确的箔段落对。我们计算每个指标的准确性，即为正确的候选段落相对于箔分配更高的分数。To test the sensitivity of EMScore in identifying foil captions containing inaccurate visual concepts, we followed the paradigm of the foil-COCO dataset to modify the ActivityNet-Entities test dataset and construct the ActivityNet-Foil dataset. In ActivityNet-Entities, each video has two corresponding paragraphs. We used one of the two paragraphs to construct the correct foil pair, and the other as the reference for reference-based metrics. Each paragraph contains approximately 3 sentences with distinct timestamps, and the visual concepts in each sentence are grounded in annotated bounding boxes. Foil captions are created by replacing the original visual concepts with similar but incorrect ones. Our data generation process has three main steps: First, we collect all visual concepts and filter out low-frequency ones. Then, we pair words belonging to the same supercategory (e.g., river-pool, shirt-shoe, catdog). Finally, we obtain 2,191 correct foil pairs, with approximately 13 foil pairs per visual concept. Second, we replace the visual concepts in the original correct captions with the paired foil candidates to build candidate foil captions. Each correct caption has multiple candidate foil captions. Third, for each correct caption, we mine the most challenging foil caption by selecting the candidate with the lowest perplexity. Finally, we create 1,900 correct foil paragraph pairs, where at least one caption in each foil paragraph contains foil visual concepts. As shown in Figure 4, each video contains one correct foil paragraph pair. We calculate the accuracy of each metric, which is defined as assigning a higher score to the correct candidate paragraph relative to the foil one.

搜集汇总

背景与挑战

背景概述

ActivityNet-FOIL是一个基于ActivityNet-Entities测试数据集构建的数据集，旨在评估EMScore对识别包含错误视觉概念的箔片标题的敏感性。通过替换原始视觉概念为类似但错误的候选词，生成了1900个正确-箔片段落对，每个视频包含一对，用于计算指标准确性。该数据集于2022年由多所中国高校和机构发布，相关资源可在GitHub和CVPR论文中获取。