FINEBENCH/FineBench

--- license: mit --- # Dataset Card for FineBench FineBench is a large-scale, multiple-choice Video Question Answering (VQA) dataset designed specifically to evaluate the fine-grained understanding of human actions in videos. It leverages the dense spatial (bounding boxes) and temporal (timestamps) annotations from the AVA v2.2 dataset, providing ~200k questions focused on nuanced person movements, interactions, and object manipulations within long video contexts. ## Dataset Details ### Dataset Description FineBench addresses a key gap in existing VQA benchmarks by focusing on **fine-grained human action understanding** coupled with **dense spatio-temporal grounding**. Based on the AVA v2.2 dataset, which annotates atomic visual actions in movie clips, FineBench automatically generates multiple-choice questions (MCQs) using a template-based approach. Each question probes specific aspects of person movement, person interaction, or object manipulation, referencing individuals using spatial descriptors derived from their bounding boxes. The dataset includes ~200k QA pairs across 64 unique source videos (derived from AVA sources, primarily movies), with an average video duration of 900 seconds and high QA density. Its primary goal is to provide a challenging benchmark for evaluating the ability of Vision-Language Models (VLMs) to precisely localize and comprehend subtle human behaviors in complex scenes over time. - **Curated by:** N/A - **Language(s) (NLP):** English - **License:** MIT ### Dataset Sources - **Repository:** https://huggingface.co/datasets/FINEBENCH/FineBench - **Paper:** Coming Soon - **Demo:** Coming Soon ## Uses ### Direct Use FineBench is primarily intended for **evaluating and benchmarking Vision-Language Models (VLMs)** on tasks requiring fine-grained understanding of human actions in videos. Specific use cases include: * Assessing model capabilities in spatio-temporal reasoning regarding human actions. * Evaluating understanding of nuanced person movement, person interaction, and object manipulation categories. * Probing model robustness in handling multiple actors and spatial references within complex scenes. * Analyzing model failure modes related to fine-grained comprehension (as demonstrated in the associated paper). * [Stretch] Training or fine-tuning VLMs to improve fine-grained action understanding (though primarily designed as a benchmark). ### Out-of-Scope Use FineBench is **not suitable** for: * Directly inferring real-world statistics about human behavior (due to the source videos being primarily movies). * Training models for surveillance or sensitive identity recognition, as it lacks the necessary labels and focuses on atomic actions from fictional content. Misuse related to analyzing depicted sensitive actions, even if fictional, should be avoided. ## Dataset Structure FineBench is structured as a multiple-choice question-answering dataset. Each instance typically corresponds to a question about a specific person within a specific timestamped segment of a video. The key fields likely include: * `video_id`: Identifier for the source video. * `timestamp`: Timestamp indicating the relevant moment or segment in the video. * `bbox`: Bounding box coordinates for the person(s) relevant to the question. * `question`: The generated multiple-choice question (string). * `options`: A list of possible answers (strings), including the correct answer and generated distractors. * `answer`: The index of the correct answer within the `options` list. * `action_name`: The ground truth action label(s) the question is based on. * `action_type`: The high-level category (Person Movement, Person Interaction, Object Manipulation) the question pertains to. The dataset structure ensures that each question is grounded in specific spatial regions (bounding boxes) and temporal moments (timestamps). ## Dataset Creation ### Curation Rationale Existing VQA datasets often lack the necessary dense spatial and temporal grounding, or the specific focus on fine-grained human actions required to rigorously evaluate modern VLMs' capabilities in nuanced video understanding. As shown in analyses accompanying this dataset, even state-of-the-art VLMs struggle with precisely localizing actions and distinguishing between subtle variations in human movement and interaction. FineBench was created to directly address this gap, providing a large-scale, challenging benchmark specifically designed to probe these fine-grained understanding abilities. ### Source Data The primary source data for FineBench is the **AVA (Atomic Visual Actions) v2.2 dataset** \cite{gu2018ava}. AVA provides dense annotations of atomic visual actions performed by humans within movie clips, including: * Action labels (80 atomic actions). * Bounding boxes localizing the person performing the action. * Timestamps indicating when the action occurs. FineBench utilizes these annotations and the corresponding video segments from AVA's source movies. #### Data Collection and Processing The FineBench QA pairs were not manually collected but **algorithmically generated** based on the AVA v2.2 annotations. The process involved: 1. **Template-Based Question Generation:** A comprehensive set of question templates (~70) was designed, categorized by action type (Person Movement, Object Manipulation, Person Interaction). 2. **Spatial Referencing:** Placeholders in templates (e.g., `{person}`) were instantiated using dynamic spatial descriptors (e.g., "the leftmost person", "the person in the center", "the second person from the left") derived from AVA bounding box locations to ensure unambiguous subject reference. 3. **Distractor Selection:** For each question based on a ground truth AVA action, plausible incorrect answer options (distractors) were selected using a two-tiered strategy: first prioritizing semantically similar actions based on a predefined mapping, and falling back to random selection within the same action category if necessary. Compound questions were generated for simultaneous actions. 4. **Data Structuring:** The generated questions, options, correct answer labels, and relevant metadata (video ID, timestamp, bounding box, action category) were compiled into the final dataset splits, preserving the original AVA annotations. #### Who are the source data producers? The original annotations (action labels, bounding boxes, timestamps) were created by human annotators as part of the AVA v2.2 dataset curation process \cite{gu2018ava}. Details on the annotators (demographics, compensation) are available in the original AVA publications. The underlying visual data comes from movies, produced by various film studios, directors, actors, etc. #### Annotation process Described in the accompanying paper. #### Who are the annotators? * **Base Annotations (Actions, Boxes):** Human annotators for AVA v2.2. * **QA Pairs (Questions, Distractors):** Algorithmically generated by the creators of FineBench ([N/A]). #### Personal and Sensitive Information The source videos are from commercially distributed movies, not private recordings. Therefore, the risk of exposing PII of individuals in the traditional sense is low. The dataset itself does not contain explicit PII beyond potentially identifiable actors (who are public figures). No anonymization was applied as the source material is public-domain or commercially distributed film content. However, the *actions* depicted (even if fictional) could potentially be sensitive depending on the context (e.g., depictions of violence, specific interactions). ## Bias, Risks, and Limitations * **Bias:** FineBench inherits potential biases from its source, AVA v2.2, which is based on movies. * **Limitations:** * Focuses exclusively on human actions; does not cover general scene understanding or object-centric VQA beyond human manipulation. ## Citation **BibTeX:** ```bibtex Coming Soon--- 许可证：MIT --- # FineBench数据集卡片 FineBench是一个大规模多项选择视频问答（Video Question Answering, VQA）数据集，专为评估对视频中人类动作的细粒度理解而设计。该数据集利用了AVA v2.2数据集的密集空间（边界框，bounding box）和时间（时间戳，timestamp）标注，提供了约20万个问题，聚焦于长视频场景中细微的人物动作、交互与物体操作。 ## 数据集详情 ### 数据集描述 FineBench弥补了现有VQA基准测试的关键空白，专注于**细粒度人类动作理解**与**密集时空定位**的结合。基于AVA v2.2数据集（该数据集对电影片段中的原子视觉动作进行标注），FineBench通过基于模板的方法自动生成多项选择题（Multiple-Choice Questions, MCQs）。每个问题都会探究人物动作、人物交互或物体操作的特定维度，并通过从边界框派生的空间描述符来指代特定个体。该数据集包含来自64个原始视频（源自AVA数据集，主要为电影）的约20万个问答对，平均视频时长为900秒，问答密度极高。其核心目标是提供一个具有挑战性的基准测试，用于评估视觉语言模型（Vision-Language Models, VLMs）在复杂场景中随时间精确定位并理解细微人类行为的能力。 - **数据整理方：** 未公开 - **语言（自然语言处理）：** 英语 - **许可证：** MIT ## 数据集来源 - **仓库地址：** https://huggingface.co/datasets/FINEBENCH/FineBench - **论文：** 即将发布 - **演示：** 即将发布 ## 使用场景 ### 直接使用 FineBench主要用于**评估视觉语言模型（Vision-Language Models, VLMs）**，针对需要理解视频中人类动作细粒度信息的任务。具体应用场景包括： * 评估模型在人类动作相关时空推理方面的能力 * 评估模型对细微人物动作、人物交互与物体操作类别的理解能力 * 测试模型在复杂场景中处理多个角色与空间引用的鲁棒性 * 分析与细粒度理解相关的模型失效模式（详见配套论文） * [拓展场景] 训练或微调视觉语言模型以提升细粒度动作理解能力（尽管其主要设计目标为基准测试） ### 超出范围的使用 FineBench**不适用于**： * 直接推断人类行为的真实世界统计数据（由于原始视频主要为电影素材） * 训练用于监控或敏感身份识别的模型，因为该数据集缺乏必要的标签，且仅聚焦于虚构内容中的原子动作。应避免任何与分析所描绘敏感动作相关的滥用行为，即使这些动作是虚构的。 ## 数据集结构 FineBench采用多项选择问答数据集的结构。每个样本通常对应一个关于视频中特定时间片段内特定人物的问题。其核心字段通常包括： * `video_id`：原始视频的标识符 * `timestamp`：指示视频中相关时刻或片段的时间戳 * `bbox`：与问题相关的人物的边界框坐标 * `question`：生成的多项选择题（字符串类型） * `options`：包含正确答案与干扰项的候选答案列表（字符串类型） * `answer`：`options`列表中正确答案的索引 * `action_name`：该问题所基于的真实动作标签（多个） * `action_type`：该问题所属的高级类别（人物动作、人物交互、物体操作） 该数据集结构确保每个问题都锚定在特定的空间区域（边界框）与时间节点（时间戳）上。 ## 数据集构建 ### 构建依据 现有VQA数据集往往缺乏必要的密集时空定位能力，或未聚焦于细粒度人类动作，无法严格评估现代视觉语言模型在细微视频理解方面的能力。正如本数据集配套分析所示，即使是当前最先进的视觉语言模型，在精确定位动作与区分人类动作与交互的细微差异方面仍存在困难。FineBench的构建正是为了弥补这一空白，提供一个大规模、具有挑战性的基准测试，专门用于探测这些细粒度理解能力。 ### 源数据 FineBench的主要源数据为**AVA（原子视觉动作，Atomic Visual Actions）v2.2数据集**cite{gu2018ava}。AVA数据集对电影片段中人类执行的原子视觉动作进行密集标注，包括： * 动作标签（共80个原子动作） * 定位执行动作人物的边界框 * 指示动作发生时间的时间戳 FineBench利用了这些标注以及AVA源电影中的对应视频片段。 #### 数据收集与处理 FineBench的问答对并非手动收集，而是**基于AVA v2.2标注通过算法生成**。具体流程包括： 1. **基于模板的问题生成**：设计了约70个全面的问题模板，按动作类型（人物动作、物体操作、人物交互）进行分类。 2. **空间引用实例化**：模板中的占位符（如`{person}`）将通过动态空间描述符（如"最左侧的人物"、"中心位置的人物"、"从左数第二个人物"）进行填充，这些描述符源自AVA的边界框位置，以确保指代对象的明确性。 3. **干扰项选择**：对于每个基于真实AVA动作的问题，将采用两级策略选择合理的错误答案选项（干扰项）：首先优先基于预定义映射选择语义相似的动作，若无法满足则在同一动作类别内随机选择。针对同时发生的动作，将生成复合问题。 4. **数据结构化**：将生成的问题、选项、正确答案标签以及相关元数据（视频ID、时间戳、边界框、动作类别）整合为最终的数据集划分，保留原始AVA标注。 #### 源数据生产者是谁？ 原始标注（动作标签、边界框、时间戳）由人类标注员在AVA v2.2数据集的构建过程中完成cite{gu2018ava}。有关标注员的详细信息（人口统计、薪酬）可参阅原始AVA论文。原始视觉数据来自电影，由各类电影工作室、导演、演员等制作。 #### 标注流程 详见配套论文。 #### 标注人员是谁？ * **基础标注（动作、边界框）：** AVA v2.2数据集的人类标注员 * **问答对（问题、干扰项）：** 由FineBench的创建者通过算法生成（[未公开]）。 #### 个人与敏感信息 源视频来自商业发行的电影，而非私人录制内容。因此，传统意义上暴露个人可识别信息（Personally Identifiable Information, PII）的风险较低。该数据集本身未包含明确的个人可识别信息，仅可能涉及可识别的公众人物演员。由于源素材为公共领域或商业发行的电影内容，未进行匿名化处理。然而，所描绘的动作（即使是虚构的）可能根据具体语境构成敏感内容（例如暴力描绘、特定交互）。 ## 偏差、风险与局限性 * **偏差：** FineBench继承了其源数据集AVA v2.2的潜在偏差，而AVA v2.2基于电影素材。 * **局限性：** * 仅聚焦于人类动作，未涵盖通用场景理解或超出人类操作范畴的以物体为中心的VQA任务。 ## 引用 **BibTeX格式：** bibtex 即将发布