hrinnnn/PerceptionComp

--- pretty_name: PerceptionComp license: other task_categories: - video-question-answering - multiple-choice language: - en tags: - video - benchmark - multimodal - reasoning - video-understanding - evaluation size_categories: - 1K<n<10K --- # PerceptionComp PerceptionComp is a benchmark for complex perception-centric video reasoning. It focuses on questions that cannot be solved from a single frame, a short clip, or a shallow caption. Models must revisit visually complex videos, gather evidence across temporally separated segments, and combine multiple perceptual cues before answering. ## Dataset Details ### Dataset Description PerceptionComp contains 1,114 manually annotated five-choice questions associated with 273 videos. The benchmark covers seven categories: outdoor tour, shopping, sport, variety show, home tour, game, and movie. This Hugging Face dataset repository hosts the benchmark videos. The official annotation file, evaluation code, and model integration examples are maintained in the GitHub repository: - GitHub repository: https://github.com/hrinnnn/PerceptionComp - **Curated by:** PerceptionComp authors - **Language(s) (NLP):** English - **License:** Please replace `other` in the metadata above with the final data license before public release if a more specific license applies. ### Dataset Sources - **Repository:** https://github.com/hrinnnn/PerceptionComp <!-- - **Paper:** Add the public paper link here when available. --> ## Uses ### Direct Use PerceptionComp is intended for: - benchmarking video-language models on complex perception-centric reasoning - evaluating long-horizon and multi-evidence video understanding - comparing proprietary and open-source multimodal models under a unified protocol Users are expected to download the videos from this Hugging Face dataset and run evaluation with the official GitHub repository. ### Out-of-Scope Use PerceptionComp is not intended for: - unrestricted commercial redistribution of hosted videos when original source terms do not allow it - surveillance, identity inference, or sensitive attribute prediction - modifying the benchmark protocol and reporting those results as directly comparable official scores ## Dataset Structure ### Data Instances Each benchmark question is associated with: - one `video_id` - one multiple-choice question - five answer options - one correct answer - one semantic category - one difficulty label The official annotation file is maintained in the GitHub repository: - `benchmark/annotations/1-1114.json` Core fields in each annotation item: - `key`: question identifier - `video_id`: video filename stem without `.mp4` - `question`: question text - `answer_choice_0` to `answer_choice_4`: five answer options - `answer_id`: zero-based index of the correct option - `answer`: text form of the correct answer - `category`: semantic category - `difficulty`: difficulty label ### Data Files The Hugging Face dataset stores the benchmark videos. The official evaluation code prepares them into the following local layout: ```text benchmark/videos/<video_id>.mp4 ``` Use the official download script from the GitHub repository: ```bash git clone https://github.com/hrinnnn/PerceptionComp.git cd PerceptionComp pip install -r requirements.txt python scripts/download_data.py --repo-id hrinnnn/PerceptionComp ``` ### Data Splits The current public release uses one official evaluation set: - `1-1114.json`: 1,114 multiple-choice questions over 273 videos ## Dataset Creation ### Curation Rationale PerceptionComp was created to evaluate a failure mode that is not well covered by simpler video benchmarks: questions that require models to combine multiple perceptual constraints over time instead of relying on a single salient frame or a short summary. ### Source Data The benchmark uses real-world videos paired with manually written multiple-choice questions. #### Data Collection and Processing Videos were collected and organized for benchmark evaluation. Annotation authors then wrote perception-centric multiple-choice questions for the selected videos. Each question was designed to require visual evidence from the video rather than simple prior knowledge or caption-level shortcuts. The release process includes: - associating each question with a `video_id` - formatting each sample as a five-choice multiple-choice item - assigning semantic categories - assigning difficulty labels - consolidating the release into one official annotation file #### Who are the source data producers? The underlying videos may originate from third-party public sources. The benchmark annotations were created by the PerceptionComp authors and collaborators. ### Annotations #### Annotation Process PerceptionComp contains 1,114 manually annotated five-choice questions. Questions were written to test perception-centric reasoning over videos rather than single-frame recognition alone. #### Who are the annotators? The annotations were created by the PerceptionComp project team. #### Personal and Sensitive Information The videos may contain people, faces, voices, public scenes, or other naturally occurring visual content. The dataset is intended for research evaluation, not for identity inference or sensitive attribute prediction. ### Recommendations Users should: - report results with the official evaluation code - avoid changing prompts, parsing rules, or metrics when claiming benchmark numbers - verify that their usage complies with the terms of the original video sources - avoid using the dataset for surveillance, identity recognition, or sensitive attribute inference ## Citation If you use PerceptionComp, please cite the project paper when it is publicly available. ```bibtex @misc{perceptioncomp2026, title={PerceptionComp}, author={PerceptionComp Authors}, year={2026}, howpublished={Hugging Face dataset and GitHub repository} } ``` ## More Information Official evaluation code and documentation: - GitHub: https://github.com/hrinnnn/PerceptionComp Example evaluation workflow: ```bash git clone https://github.com/hrinnnn/PerceptionComp.git cd PerceptionComp pip install -r requirements.txt python scripts/download_data.py --repo-id hrinnnn/PerceptionComp python evaluate/evaluate.py \ --model YOUR_MODEL_NAME \ --provider api \ --api-key YOUR_API_KEY \ --base-url YOUR_BASE_URL \ --video-dir benchmark/videos ``` ## Dataset Card Authors PerceptionComp authors--- pretty_name: PerceptionComp 许可证: other 任务类别: - 视频问答（video-question-answering） - 多项选择（multiple-choice） 语言: - en 标签: - 视频 - 基准数据集（benchmark） - 多模态（multimodal） - 推理（reasoning） - 视频理解（video-understanding） - 评估（evaluation） 规模类别: - 1K<n<10K --- # PerceptionComp PerceptionComp是一款面向复杂感知导向型视频推理的基准数据集。其聚焦于无法通过单帧图像、短片段或浅层字幕求解的问题，要求模型回溯视觉复杂度较高的视频，从时间上分离的多个片段中收集证据，并结合多种感知线索后再给出答案。 ## 数据集详情 ### 数据集描述 PerceptionComp包含1114条经人工标注的五选项多项选择题，关联273个视频。该基准数据集涵盖7大类场景：户外游览、购物、体育赛事、综艺、家居游览、游戏及电影。 本Hugging Face数据集仓库托管该基准数据集的视频文件，官方标注文件、评估代码及模型集成示例均维护于GitHub仓库： - GitHub仓库：https://github.com/hrinnnn/PerceptionComp - **数据集维护方：** PerceptionComp作者团队 - **自然语言处理所用语言：** 英语 - **许可协议：** 若有更具体的许可协议适用，请在公开发布前将上述元数据中的`other`替换为最终的数据许可协议。 ### 数据集来源 - **代码仓库：** https://github.com/hrinnnn/PerceptionComp <!-- - **论文：** 公开论文链接将在可用时补充。 --> ## 使用场景 ### 直接使用场景 PerceptionComp旨在用于： - 面向复杂感知导向型推理的视频语言模型基准测试 - 评估长时序多证据视频理解能力 - 在统一协议下对比闭源与开源多模态模型 用户需从本Hugging Face数据集下载视频，并通过官方GitHub仓库运行评估流程。 ### 不适宜使用场景 PerceptionComp不适用于： - 当原始视频源条款不允许时，无限制地商业重分发托管视频 - 监控、身份推断或敏感属性预测 - 修改基准测试协议并将所得结果报告为可直接对比的官方评分 ## 数据集结构 ### 数据实例 每个基准测试问题关联以下内容： - 一个`video_id`（视频标识符） - 一个多项选择题 - 五个答案选项 - 一个正确答案 - 一个语义类别 - 一个难度标签 官方标注文件维护于GitHub仓库的路径：`benchmark/annotations/1-1114.json` 每个标注项的核心字段包括： - `key`：问题标识符 - `video_id`：不含`.mp4`后缀的视频文件名主干 - `question`：问题文本 - `answer_choice_0`至`answer_choice_4`：五个答案选项 - `answer_id`：正确选项的从零开始的索引 - `answer`：正确答案的文本形式 - `category`：语义类别 - `difficulty`：难度标签 ### 数据文件 本Hugging Face数据集存储基准数据集的视频文件。官方评估代码会将其整理为如下本地目录结构： text benchmark/videos/<video_id>.mp4 请使用GitHub仓库中的官方下载脚本： bash git clone https://github.com/hrinnnn/PerceptionComp.git cd PerceptionComp pip install -r requirements.txt python scripts/download_data.py --repo-id hrinnnn/PerceptionComp ### 数据划分 当前公开版本使用一套官方评估集： - `1-1114.json`：涵盖273个视频的1114条多项选择题 ## 数据集创建 ### 设计初衷 PerceptionComp的创建旨在评估现有简易视频基准数据集未充分覆盖的一类失效模式：即要求模型结合时间维度上的多种感知约束条件，而非依赖单个显著帧或简短摘要求解的问题。 ### 源数据 本基准数据集使用真实世界视频搭配人工编写的多项选择题。 #### 数据收集与处理 视频经收集整理后用于基准测试评估，标注人员为所选视频编写感知导向型多项选择题。每道问题的设计均要求依赖视频中的视觉证据作答，而非依赖简单的先验知识或字幕层面的捷径。 发布流程包括： - 为每个问题关联`video_id` - 将每个样本格式化为五选项多项选择题条目 - 分配语义类别 - 分配难度标签 - 整合为一份官方标注文件后发布 #### 源数据生产者是谁？ 原始视频可能来源于第三方公开渠道。本基准数据集的标注由PerceptionComp作者团队及合作者完成。 ### 标注信息 #### 标注流程 PerceptionComp包含1114条经人工标注的五选项多项选择题。题目设计用于测试视频层面的感知导向型推理能力，而非仅单帧识别能力。 #### 标注人员是谁？ 标注工作由PerceptionComp项目团队完成。 #### 个人与敏感信息 视频可能包含人物、面部、语音、公共场景或其他自然出现的视觉内容。本数据集仅用于研究评估，不得用于身份推断或敏感属性预测。 ## 使用建议 用户应： - 使用官方评估代码报告实验结果 - 当声明基准测试性能时，不得修改提示词、解析规则或评估指标 - 确认其使用行为符合原始视频源的许可条款 - 不得将本数据集用于监控、身份识别或敏感属性推断 ## 引用规范 若您使用PerceptionComp，请在该项目论文公开后引用该论文。 bibtex @misc{perceptioncomp2026, title={PerceptionComp}, author={PerceptionComp Authors}, year={2026}, howpublished={Hugging Face dataset and GitHub repository} } ## 更多信息 官方评估代码及文档： - GitHub：https://github.com/hrinnnn/PerceptionComp 示例评估流程： bash git clone https://github.com/hrinnnn/PerceptionComp.git cd PerceptionComp pip install -r requirements.txt python scripts/download_data.py --repo-id hrinnnn/PerceptionComp python evaluate/evaluate.py --model YOUR_MODEL_NAME --provider api --api-key YOUR_API_KEY --base-url YOUR_BASE_URL --video-dir benchmark/videos ## 数据集卡片作者 PerceptionComp作者团队