yayuanli/MATT-Bench

--- license: cc-by-4.0 task_categories: - video-classification - video-text-to-text - object-detection tags: - egocentric-video - mistake-detection - temporal-localization - video-language-grounding - hand-object-interaction - action-recognition - procedural-activities - semantic-role-labeling - ego4d - epic-kitchens - point-of-no-return - cvpr2026 pretty_name: MATT-Bench size_categories: - 100K<n<1M --- # Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos **CVPR 2026** [Yayuan Li](https://www.linkedin.com/in/yayuan-li-148659272/)¹, [Aadit Jain](https://www.linkedin.com/in/jain-aadit/)¹, [Filippos Bellos](https://www.linkedin.com/in/filippos-bellos-168595156/)¹, [Jason J. Corso](https://www.linkedin.com/in/jason-corso/)^1,2 ¹University of Michigan, ²Voxel51 [[Paper](https://arxiv.org/abs/2511.20525)] [[Code](https://github.com/yayuanli/MATT)] [[Project Page](https://yayuanli.github.io/MATT/)] --- > **Dataset coming soon.** We are preparing the data for public release. Stay tuned! ## MATT-Bench Overview MATT-Bench provides two large-scale benchmarks for **Mistake Attribution (MATT)** — a task that goes beyond binary mistake detection to attribute *what* semantic role was violated, *when* the mistake became irreversible (Point-of-No-Return), and *where* the mistake occurred in the frame. The benchmarks are constructed by **MisEngine**, a data engine that automatically creates mistake samples with attribution-rich annotations from existing egocentric action datasets: | Dataset | Samples | Instruction Texts | Semantic | Temporal | Spatial | |---|---|---|---|---|---| | **Ego4D-M** | 257,584 | 16,099 | ✓ | ✓ | ✓ | | **EPIC-KITCHENS-M** | 221,094 | 12,283 | ✓ | — | — | These are at least **two orders of magnitude larger** than any existing mistake dataset. ## Annotations Each sample consists of an instruction text and an attempt video, annotated with: - **Semantic Attribution**: Which semantic role (predicate, object) in the instruction is violated in the attempt video - **Temporal Attribution**: The Point-of-No-Return (PNR) frame where the mistake becomes irreversible (Ego4D-M) - **Spatial Attribution**: Bounding box localizing the mistake region in the PNR frame (Ego4D-M) ## Citation ```bibtex @inproceedings{li2026mistakeattribution, title = {Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos}, author = {Li, Yayuan and Jain, Aadit and Bellos, Filippos and Corso, Jason J.}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2026}, } ```license: 知识共享署名4.0（CC BY 4.0） task_categories: - 视频分类 - 视频-文本到文本 - 目标检测 tags: - 第一人称视频（egocentric video） - 错误检测 - 时序定位 - 视频语言接地（video-language-grounding） - 手-物体交互（hand-object-interaction） - 动作识别 - 程序性活动 - 语义角色标注 - Ego4D - EPIC-KITCHENS - 临界点（Point-of-No-Return） - CVPR2026 pretty_name: MATT-Bench size_categories: - 10万 < 样本量 < 100万 --- # 错误归因：第一人称视频中的细粒度错误理解 **CVPR 2026** [Yayuan Li](https://www.linkedin.com/in/yayuan-li-148659272/)¹、[Aadit Jain](https://www.linkedin.com/in/jain-aadit/)¹、[Filippos Bellos](https://www.linkedin.com/in/filippos-bellos-168595156/)¹、[Jason J. Corso](https://www.linkedin.com/in/jason-corso/)^1,2 ¹密歇根大学, ²Voxel51 [[论文](https://arxiv.org/abs/2511.20525)] [[代码](https://github.com/yayuanli/MATT)] [[项目主页](https://yayuanli.github.io/MATT/)] --- > **数据集即将发布。我们正在筹备数据的公开发布工作，敬请期待！** ## MATT-Bench 概述 MATT-Bench 为**错误归因（Mistake Attribution, MATT）**任务提供两个大规模基准测试集——该任务突破了二元错误检测的局限，可对*被违反的语义角色*、*错误变为不可逆转的临界点（Point-of-No-Return）*以及*错误在帧中的发生位置*进行细粒度归因。 该基准测试集由**MisEngine**构建，这是一款可从现有第一人称动作数据集中自动生成带有丰富归因标注的错误样本的数据引擎。 | 数据集 | 样本量 | 指令文本数量 | 语义标注 | 时序标注 | 空间标注 | |--------------|------------|--------------|----------|----------|----------| | **Ego4D-M** | 257,584 | 16,099 | ✓ | ✓ | ✓ | | **EPIC-KITCHENS-M** | 221,094 | 12,283 | ✓ | — | — | 上述基准测试集的规模至少比现有所有错误数据集高出两个数量级。 ## 标注 每个样本均包含一条指令文本与一段尝试视频，并标注以下三类信息： - **语义归因**：指令文本中的哪些语义角色（谓词、对象）在尝试视频中被违反 - **时序归因**：错误变为不可逆转的临界点（Point-of-No-Return, PNR）帧（仅适用于Ego4D-M） - **空间归因**：用于定位PNR帧中错误区域的边界框（仅适用于Ego4D-M） ## 引用 bibtex @inproceedings{li2026mistakeattribution, title = {Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos}, author = {Li, Yayuan and Jain, Aadit and Bellos, Filippos and Corso, Jason J.}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2026}, }