mistakeattribution/MATT-Bench

--- license: cc-by-4.0 task_categories: - video-classification - video-text-to-text - object-detection tags: - egocentric-video - mistake-detection - temporal-localization - video-language-grounding - hand-object-interaction - action-recognition - procedural-activities - semantic-role-labeling - ego4d - epic-kitchens - holoassist - point-of-no-return - cvpr2026 pretty_name: MATT-Bench size_categories: - 100K<n<1M configs: - config_name: ego4d data_files: - split: train path: ego4d/parquet/train.parquet - split: valid path: ego4d/parquet/valid.parquet - split: test path: ego4d/parquet/test.parquet - config_name: epickitchens data_files: - split: train path: epickitchens/parquet/train.parquet - split: validation path: epickitchens/parquet/validation.parquet - config_name: holoassist data_files: - split: train path: holoassist/parquet/train.parquet - split: validation path: holoassist/parquet/validation.parquet --- # Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos **CVPR 2026** [Yayuan Li](https://www.linkedin.com/in/yayuan-li-148659272/)¹, [Aadit Jain](https://www.linkedin.com/in/jain-aadit/)¹, [Filippos Bellos](https://www.linkedin.com/in/filippos-bellos-168595156/)¹, [Jason J. Corso](https://www.linkedin.com/in/jason-corso/)^1,2 ¹University of Michigan, ²Voxel51 [[Paper](https://arxiv.org/abs/2511.20525)] [[Code](https://github.com/yayuanli/MATT)] [[Project Page](https://yayuanli.github.io/MATT/)] --- ## MATT-Bench Overview MATT-Bench provides large-scale benchmarks for **Mistake Attribution (MATT)** — a task that goes beyond binary mistake detection to attribute *what* semantic role was violated, *when* the mistake became irreversible (Point-of-No-Return), and *where* the mistake occurred in the frame. The benchmarks are constructed by **MisEngine**, a data engine that automatically creates mistake samples with attribution-rich annotations from existing egocentric action datasets: | Dataset | Samples | Instruction Texts | Semantic | Temporal | Spatial | |---------------------|---------|-------------------|----------|----------|---------| | **Ego4D-M** | 220,800 | 19,467 | ✓ | ✓ | ✓ | | **EPIC-KITCHENS-M** | 299,715 | 12,283 | ✓ | — | — | These are at least **two orders of magnitude larger** than any existing mistake dataset. Instruction-text counts = unique (predicate `V`, argument `ARG1`) pairs. A third source, **HoloAssist-M**, is released alongside as an additional benchmark — see [Extended: HoloAssist-M](#extended-holoassist-m) below. **Repository Layout** ``` MATT-Bench/ ├── ego4d/ │ ├── train.xlsx, valid.xlsx, test.xlsx ← primary annotation files (consumed by the MATT codebase) │ ├── parquet.xlsx ← MisEngine reproduction data (Ego4D narrations with SRL) │ └── parquet/ ← Parquet mirror for the HF dataset viewer ├── epickitchens/ │ ├── train.xlsx, validation.xlsx │ └── parquet/ └── holoassist/ ├── train.xlsx, validation.xlsx └── parquet/ ``` `.xlsx` is the canonical download format (the MATT codebase reads Excel directly). The `parquet/` mirror powers the HF dataset viewer and `datasets.load_dataset(...)` loaders — both views contain the same rows. ## Downloading MATT-Bench MATT-Bench has two parts that you obtain separately: 1. **Annotations** — semantic attribution annotations are hosted here, download via `hf` or `git clone`. Temporal and spatial attribution annotations are inherited from the original dataset. 2. **Video media** — **not** hosted here. Download from each source dataset using the instructions below. Original videos retain their upstream licenses. ### Annotations (this repo) ```bash # Everything hf download mistakeattribution/MATT-Bench --repo-type dataset --local-dir MATT-Bench # Just one source dataset's xlsx files hf download mistakeattribution/MATT-Bench --repo-type dataset \ --include "ego4d/*.xlsx" --local-dir MATT-Bench ``` Or via the `datasets` library (reads the parquet mirror): ```python from datasets import load_dataset ego4d_m = load_dataset("mistakeattribution/MATT-Bench", "ego4d") epic_m = load_dataset("mistakeattribution/MATT-Bench", "epickitchens") holo_m = load_dataset("mistakeattribution/MATT-Bench", "holoassist") ``` ### Video media #### Ego4D Follow <https://ego4d-data.org/docs/CLI/> to download. The `video_uid` and `clip1_uid` fields in our annotations correspond to Ego4D's native video and clip UIDs. MATT-Bench uses the FHO (Forecasting Hands and Objects) benchmark clips from Ego4D. Example downloading script: ```bash ego4d --output_directory="~/ego4d_data" --datasets clips --benchmarks FHO ``` #### EPIC-KITCHENS-100 Follow <https://epic-kitchens.github.io/> to download. MATT-Bench's `video_id` matches EPIC's participant-video identifier (e.g. `P22_16`); `start_frame` / `end_frame` index the RGB frame sequence. Example download script: ```bash git clone https://github.com/epic-kitchens/epic-kitchens-download-scripts cd epic-kitchens-download-scripts python epic_downloader.py --rgb-frames # or --videos ``` #### HoloAssist Although not reported in the paper, we also support the HoloAssist dataset. Download the following from the [HoloAssist project page](https://holoassist.github.io/): | Resource | Link | Size | |------------------------|------------------------------------------------------------------------------------------------------------------------------------------|-----------| | Videos (pitch-shifted) | [video_pitch_shifted.tar](https://hl2data.z5.web.core.windows.net/holoassist-data-release/video_pitch_shifted.tar) | 184.20 GB | | Labels | [data-annotation-trainval-v1_1.json](https://hl2data.z5.web.core.windows.net/holoassist-data-release/data-annotation-trainval-v1_1.json) | 111 MB | | Dataset splits | [data-splits-v1_2.zip](https://holoassist.github.io/label_files/data-splits-v1_2.zip) | — | MATT-Bench's `video_id` matches HoloAssist's video identifier (e.g. `R076-21July-DSLR`). ## Data Schema ### `ego4d/{train,valid,test}.xlsx` — 13 columns | Column | Description | |-----------------------------------------------------|----------------------------------------------------------------------------------------| | `video_uid` | Ego4D video UID (full video) | | `start_frame`, `end_frame` | Frame bounds of the attempt clip | | `clip1_uid`, `clip1_start_frame`, `clip1_end_frame` | Primary Ego4D clip | | `clip2_uid`, `clip2_start_frame`, `clip2_end_frame` | Some actions are distributed across two clips (`Not required` / `-1` when absent) | | `V`, `ARG1` | Predicate and argument from the instruction (e.g. `pick up`, `apple`) | | `label` | Mistake label. 0: Correct; 1: Mistaken Predicate; 2: Mistaken Object; 3: Mistaken Both | | `split` | dataset split identifier | ### `ego4d/parquet.xlsx` — 29 columns (MisEngine reproduction data) Ego4D narration-level records with semantic-role labels (`ARG0`, `V`, `ARG1`), frame/time bounds (`start_frame`/`end_frame`/`start_sec`/`end_sec`), clip-relative bounds, and noun/verb embedding vectors. Used to reproduce the MisEngine step that produces the split files above. ### `epickitchens/{train,validation}.xlsx` and `holoassist/{train,validation}.xlsx` — 8 columns | Column | Description | |----------------------------|---------------------------------------------------------| | `video_id` | Source-dataset video identifier | | `start_frame`, `end_frame` | Frame bounds of the attempt clip | | `V`, `ARG1` | Predicate and argument of the instruction text | | `label` | Mistake label | | `actual_V`, `actual_ARG1` | Predicate/argument of the action performed in the video | ### Extended: HoloAssist-M **HoloAssist-M** is an additional MATT benchmark released alongside MATT-Bench. It is **not** part of the main two-dataset evaluation reported in the CVPR 2026 paper; it uses the same MisEngine pipeline applied to the HoloAssist dataset. | Dataset | Samples | Instruction Texts | Semantic | Temporal | Spatial | |------------------|---------|-------------------|----------|----------|---------| | **HoloAssist-M** | 562,209 | 1,786 | ✓ | — | — | Schema matches EPIC-KITCHENS-M (semantic attribution only — HoloAssist does not provide native PNR frame number andb bbox annotations). ## Citation ```bibtex @inproceedings{li2026mistakeattribution, title = {Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos}, author = {Li, Yayuan and Jain, Aadit and Bellos, Filippos and Corso, Jason J.}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2026}, } ``` Please also cite the source datasets: ```bibtex @inproceedings{grauman2022ego4d, title = {Ego4D: Around the World in 3,000 Hours of Egocentric Video}, author = {Grauman, Kristen and others}, booktitle = {CVPR}, year = {2022} } @article{Damen2022RESCALING, title = {Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100}, author = {Damen, Dima and others}, journal = {IJCV}, year = {2022} } @inproceedings{wang2023holoassist, title = {HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World}, author = {Wang, Xin and others}, booktitle = {ICCV}, year = {2023} } ```license: CC-BY-4.0 任务类别： - 视频分类 - 视频到文本生成 - 目标检测 标签： - 第一人称视角视频（egocentric-video） - 错误检测 - 时序定位（temporal-localization） - 视频语言接地（video-language-grounding） - 手物交互（hand-object-interaction） - 动作识别 - 程序性活动 - 语义角色标注（semantic-role-labeling） - Ego4D - EPIC-KITCHENS - HoloAssist - 不可挽回节点（point-of-no-return） - CVPR 2026 友好名称：MATT-Bench 样本规模：100K < n < 1M 配置项： - 配置名称：ego4d 数据文件： - 拆分：训练集 路径：ego4d/parquet/train.parquet - 拆分：验证集 路径：ego4d/parquet/valid.parquet - 拆分：测试集 路径：ego4d/parquet/test.parquet - 配置名称：epickitchens 数据文件： - 拆分：训练集 路径：epickitchens/parquet/train.parquet - 拆分：验证集 路径：epickitchens/parquet/validation.parquet - 配置名称：holoassist 数据文件： - 拆分：训练集 路径：holoassist/parquet/train.parquet - 拆分：验证集 路径：holoassist/parquet/validation.parquet --- # 错误归因（Mistake Attribution, MATT）：第一人称视角视频中的细粒度错误理解 **CVPR 2026 顶会论文** [李雅媛](https://www.linkedin.com/in/yayuan-li-148659272/)¹, [阿迪特· Jain](https://www.linkedin.com/in/jain-aadit/)¹, [菲利波斯· 贝洛斯](https://www.linkedin.com/in/filippos-bellos-168595156/)¹, [杰森· J· 科索](https://www.linkedin.com/in/jason-corso/)^1,2 ¹密歇根大学, ²Voxel51公司 [[论文](https://arxiv.org/abs/2511.20525)] [[代码](https://github.com/yayuanli/MATT)] [[项目主页](https://yayuanli.github.io/MATT/)] --- ## MATT-Bench 概述 MATT-Bench 为**错误归因（Mistake Attribution, MATT）**任务提供大规模基准测试集。该任务突破二元错误检测的局限，可同时定位三类核心信息：被违反的语义角色、错误达到不可挽回节点（Point-of-No-Return）的时间点，以及错误在画面中的空间位置。 本基准测试集由**MisEngine**构建，这是一款自动化数据引擎，可从现有第一人称视角动作数据集中自动生成带有丰富归因标注的错误样本： | 数据集名称 | 样本量 | 指令文本数 | 语义归因 | 时序定位 | 空间定位 | |---------------------|---------|-------------------|----------|----------|---------| | **Ego4D-M** | 220,800 | 19,467 | ✔ | ✔ | ✔ | | **EPIC-KITCHENS-M** | 299,715 | 12,283 | ✔ | — | — | 本数据集规模较现有所有错误检测数据集至少**高出两个数量级**。指令文本数为唯一的（谓词`V`，论元`ARG1`）组合数。 第三个数据集**HoloAssist-M**作为额外基准测试集同步发布，详见下文[扩展内容：HoloAssist-M](#extended-holoassist-m)。 **仓库目录结构** MATT-Bench/ ├── ego4d/ │ ├── train.xlsx, valid.xlsx, test.xlsx ← 主标注文件（供MATT代码库直接读取） │ ├── parquet.xlsx ← MisEngine复现数据（带语义角色标注的Ego4D解说文本） │ └── parquet/ ← Hugging Face数据集查看器的Parquet格式镜像 ├── epickitchens/ │ ├── train.xlsx, validation.xlsx │ └── parquet/ └── holoassist/ ├── train.xlsx, validation.xlsx └── parquet/ `.xlsx` 为标准下载格式（MATT代码库可直接读取Excel文件）。`parquet/` 目录为Parquet格式镜像，用于支持Hugging Face数据集查看器和`datasets.load_dataset(...)`加载接口——两种格式包含完全一致的数据行。 ## MATT-Bench 下载指南 MATT-Bench 分为两个部分，需分别获取： 1. **标注文件**——语义归因标注存储于本仓库，可通过`hf`或`git clone`下载。时序与空间归因标注继承自原始数据集。 2. **视频素材**——本仓库不提供视频文件，请按照下文说明从各原始数据集下载。原始视频保留其上游许可证协议。 ### 标注文件（本仓库） bash # 下载全部数据 hf download mistakeattribution/MATT-Bench --repo-type dataset --local-dir MATT-Bench # 仅下载指定源数据集的xlsx标注文件 hf download mistakeattribution/MATT-Bench --repo-type dataset --include "ego4d/*.xlsx" --local-dir MATT-Bench 也可通过`datasets`库读取Parquet镜像数据： python from datasets import load_dataset ego4d_m = load_dataset("mistakeattribution/MATT-Bench", "ego4d") epic_m = load_dataset("mistakeattribution/MATT-Bench", "epickitchens") holo_m = load_dataset("mistakeattribution/MATT-Bench", "holoassist") ### 视频素材下载 #### Ego4D 数据集 请访问<https://ego4d-data.org/docs/CLI/>下载视频。本标注中的`video_uid`与`clip1_uid`字段对应Ego4D原生的视频与片段UID。 MATT-Bench 使用Ego4D的FHO（手物预测与目标追踪）基准测试片段。示例下载脚本： bash ego4d --output_directory="~/ego4d_data" --datasets clips --benchmarks FHO #### EPIC-KITCHENS-100 数据集 请访问<https://epic-kitchens.github.io/>下载视频。MATT-Bench中的`video_id`与EPIC的参与者-视频标识符一致（例如`P22_16`）；`start_frame`与`end_frame`用于索引RGB帧序列。 示例下载脚本： bash git clone https://github.com/epic-kitchens/epic-kitchens-download-scripts cd epic-kitchens-download-scripts python epic_downloader.py --rgb-frames # 或 --videos #### HoloAssist 数据集 尽管本文未涉及该数据集，但本基准同样支持HoloAssist数据集。 请从[HoloAssist项目主页](https://holoassist.github.io/)下载以下资源： | 资源类型 | 下载链接 | 文件大小 | |------------------------|------------------------------------------------------------------------------------------------------------------------------------------|-----------| | 音高调整后的视频 | [video_pitch_shifted.tar](https://hl2data.z5.web.core.windows.net/holoassist-data-release/video_pitch_shifted.tar) | 184.20 GB | | 标注文件 | [data-annotation-trainval-v1_1.json](https://hl2data.z5.web.core.windows.net/holoassist-data-release/data-annotation-trainval-v1_1.json) | 111 MB | | 数据集划分文件 | [data-splits-v1_2.zip](https://holoassist.github.io/label_files/data-splits-v1_2.zip) | — | MATT-Bench中的`video_id`与HoloAssist的视频标识符一致（例如`R076-21July-DSLR`）。 ## 数据格式规范 ### `ego4d/{train,valid,test}.xlsx` — 共13列 | 字段名 | 字段说明 | |-----------------------------------------------------|----------------------------------------------------------------------------------------| | `video_uid` | Ego4D完整视频的UID | | `start_frame`, `end_frame` | 尝试片段的帧范围 | | `clip1_uid`, `clip1_start_frame`, `clip1_end_frame` | 主Ego4D片段 | | `clip2_uid`, `clip2_start_frame`, `clip2_end_frame` | 部分动作分布于两个片段中（无第二片段时标注为`Not required`或`-1`） | | `V`, `ARG1` | 指令文本中的谓词与论元（例如`pick up`（拿起）、`apple`（苹果）） | | `label` | 错误标注：0为动作正确；1为谓词错误；2为对象错误；3为谓词与对象均错误 | | `split` | 数据集拆分标识符 | ### `ego4d/parquet.xlsx` — 共29列（MisEngine复现数据） 该文件为Ego4D解说级别的记录，包含语义角色标注（`ARG0`、`V`、`ARG1`）、帧/时间范围（`start_frame`/`end_frame`/`start_sec`/`end_sec`）、片段相对范围以及名词/动词嵌入向量，用于复现生成上述拆分文件的MisEngine流程。 ### `epickitchens/{train,validation}.xlsx`与`holoassist/{train,validation}.xlsx` — 共8列 | 字段名 | 字段说明 | |----------------------------|---------------------------------------------------------| | `video_id` | 原始数据集的视频标识符 | | `start_frame`, `end_frame` | 尝试片段的帧范围 | | `V`, `ARG1` | 指令文本中的谓词与论元 | | `label` | 错误标注 | | `actual_V`, `actual_ARG1` | 视频中实际执行动作的谓词与论元 | ### 扩展内容：HoloAssist-M **HoloAssist-M**是随MATT-Bench同步发布的额外基准测试集，**未纳入CVPR 2026论文中报告的双数据集主评估流程**，其采用与MATT-Bench一致的MisEngine流水线处理HoloAssist数据集。 | 数据集名称 | 样本量 | 指令文本数 | 语义归因 | 时序定位 | 空间定位 | |------------------|---------|-------------------|----------|----------|---------| | **HoloAssist-M** | 562,209 | 1,786 | ✔ | — | — | 其数据格式与EPIC-KITCHENS-M一致（仅支持语义归因——HoloAssist未提供原生的不可挽回节点帧编号与边界框标注）。 ## 引用格式 bibtex @inproceedings{li2026mistakeattribution, title = {Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos}, author = {Li, Yayuan and Jain, Aadit and Bellos, Filippos and Corso, Jason J.}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2026}, } 同时请引用以下原始数据集： bibtex @inproceedings{grauman2022ego4d, title = {Ego4D: Around the World in 3,000 Hours of Egocentric Video}, author = {Grauman, Kristen and others}, booktitle = {CVPR}, year = {2022} } @article{Damen2022RESCALING, title = {Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100}, author = {Damen, Dima and others}, journal = {IJCV}, year = {2022} } @inproceedings{wang2023holoassist, title = {HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World}, author = {Wang, Xin and others}, booktitle = {ICCV}, year = {2023} }