FraunhoferIPK/IndEgo

---
license: cc-by-4.0
task_categories:
- visual-question-answering
- summarization
- video-classification
- any-to-any
language:
- en
- de
pretty_name: IndEgo
tags:
- industrial
- egocentric
- procedural
- collaborative work
- mistake detection
- VQA
- video understanding
size_categories:
- 10K<n<100K
---

<div align="center">

# IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants

**[Vivek Chavan](https://vivekchavan.com/)¹²\*, [Yasmina Imgrund](https://www.linkedin.com/in/yasmina-imgrund/)²†, [Tung Dao](https://www.linkedin.com/in/lam-dao-tung/)²†, [Sanwantri Bai](https://www.linkedin.com/in/sanwantri-bai-0a808a1b3/)³†, [Bosong Wang](https://www.linkedin.com/in/bosong0106/)⁴†, Ze Lu⁵†, [Oliver Heimann](https://www.linkedin.com/in/oliver-heimann/)¹, [Jörg Krüger](https://www.tu.berlin/iat/ueber-uns/leitung)¹²**

<p>
¹Fraunhofer IPK, Berlin &nbsp;&nbsp; ²Technical University of Berlin &nbsp;&nbsp; ³University of Tübingen<br>
⁴RWTH Aachen University &nbsp;&nbsp; ⁵Leibniz University Hannover
</p>

*^{\*Project Lead &nbsp;&nbsp;&nbsp; †Work done during student theses/projects at Fraunhofer IPK, Berlin.}*

<div align="center">
<h3 style="display: flex; align-items: center; justify-content: center; gap: 10px; margin-top: 1em; margin-bottom: 1em;">
<img src="https://IndEgo-Dataset.github.io/assets/NeurIPS-logo.svg" alt="NeurIPS Logo" height="200">
<span>Published at NeurIPS 2025</span>
</h3>
</div>

<p>
<a href="https://IndEgo-Dataset.github.io/" target="_blank"><img src="https://img.shields.io/badge/Project-Website-blue?style=flat-square" alt="Project Website"></a>
<a href="https://openreview.net/forum?id=jKw3Qhc8m1" target="_blank"><img src="https://img.shields.io/badge/Paper-OpenReview-red?style=flat-square" alt="Paper PDF"></a>
<a href="https://github.com/Vivek9Chavan/IndEgo/" target="_blank"><img src="https://img.shields.io/badge/Code-GitHub-black?style=flat-square&logo=github" alt="Code"></a>
<a href="https://neurips.cc/virtual/2025/poster/121501" target="_blank"><img src="https://img.shields.io/badge/NeurIPS-Page-orange?style=flat-square" alt="NeurIPS Page"></a>
</p>
<p>
<a href="https://colab.research.google.com/drive/1qCZnFQNRjBuy3vBlkMy7sMTcYkTNOzgg?usp=sharing" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
</p>

</div>

---

> [!WARNING]
> # 🚧 UPDATE IN PROGRESS 🚧
>
> ### ⚠️ Based on the feedback from other community members, the dataset structure is being reorganised.
> **File paths and folder names are changing.**
>
> If you download the data right now, your local file structure may become inconsistent with future updates.
> We recommend waiting until the restructuring is complete (ETA: 12 Dec, 2025).
>
> **[👉 Click here to be notified when the dataset is ready](https://forms.gle/j8krD3At2hEWjJPg8)**

## 📖 Abstract

We introduce **IndEgo**, a multimodal **egocentric and exocentric** video dataset capturing common industrial tasks such as assembly/disassembly, logistics and organisation, inspection and repair, and woodworking. The dataset includes **3,460 egocentric recordings (~197 hours)** and **1,092 exocentric recordings (~97 hours)**.


A central focus of IndEgo is **collaborative work**, where two workers coordinate on cognitively and physically demanding tasks. The egocentric recordings include rich multimodal data — eye gaze, narration, sound, motion, and semi-dense point clouds.

We provide:
- Detailed annotations: actions, summaries, mistake labels, and narrations
- Processed outputs: eye gaze, hand poses, SLAM-based semi-dense point clouds
- Benchmarks: procedural/non-procedural task understanding, **collaborative tasks**, **Mistake Detection**, and **reasoning-based Video QA**

Baseline evaluations show that IndEgo presents a challenge for state-of-the-art multimodal models.

---

## 🧩 Citation

If you use **IndEgo** in your research, please cite our NeurIPS 2025 paper:

```bibtex
@inproceedings{Chavan2025IndEgo,
author = {Vivek Chavan and Yasmina Imgrund and Tung Dao and Sanwantri Bai and Bosong Wang and Ze Lu and Oliver Heimann and J{\"o}rg Kr{\"u}ger},
title = {IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track},
year = {2025},
url = {https://neurips.cc/virtual/2025/poster/121501}
}
```

## Acknowledgments & Funding

This work is supported by the German Federal Ministry of Research, Technology and Space (BMFTR) and the German Aerospace Center (DLR) under the KIKERP project (Grant No. 16IS23055C) within the KI4KMU program. We are grateful to the Meta AI and Reality Labs teams for the Project Aria initiative, including the research kit, associated tools, and services. We also thank Hugging Face for providing a public-dataset storage grant that enables large-scale hosting and community access to the IndEgo dataset. Data collection was conducted at the research labs and test field of the Institute of Machine Tools and Factory Management (IWF), TU Berlin. Finally, we extend our sincere thanks to all student volunteers and workers who contributed to the data collection.

<div style="display: flex; justify-content: center; align-items: center; flex-wrap: nowrap; gap: 25px; margin: 20px 0;">

<a href="https://www.bmftr.bund.de/" target="_blank">
<img src="https://raw.githubusercontent.com/IndEgo-Dataset/IndEgo-Dataset.github.io/main/assets/BMFTR_-_Logo_en.svg"
alt="BMBF" style="width:110px; height:auto;">
</a>

<a href="https://www.dlr.de/" target="_blank">
<img src="https://raw.githubusercontent.com/IndEgo-Dataset/IndEgo-Dataset.github.io/main/assets/PT_DLR_Logo_SW_D_2018_lang.png"
alt="DLR" style="width:110px; height:auto;">
</a>

<a href="https://www.tu.berlin/iwf" target="_blank">
<img src="https://raw.githubusercontent.com/IndEgo-Dataset/IndEgo-Dataset.github.io/main/assets/logo-iwf-mit-namen-en.jpg"
alt="IWF" style="width:110px; height:auto;">
</a>

<a href="https://www.projectaria.com/" target="_blank"
style="font-weight:700; font-size:0.9rem; width:110px; text-align:center;">
Meta<br>Reality Labs
</a>

<a href="https://huggingface.co/" target="_blank">
<img src="https://raw.githubusercontent.com/IndEgo-Dataset/IndEgo-Dataset.github.io/main/assets/Hf-logo-with-title.svg"
alt="Hugging Face" style="width:110px; height:auto;">
</a>

</div>

许可协议：CC BY 4.0（知识共享署名4.0）
任务类别：
- 视觉问答（Visual Question Answering, VQA）
- 摘要生成
- 视频分类
- 任意到任意（any-to-any）
语言：
- 英语
- 德语
数据集名称：IndEgo
标签：
- 工业场景
- 第一人称视角（egocentric）
- 流程化
- 协作工作
- 错误检测
- VQA（视觉问答）
- 视频理解
规模类别：10K < 样本数量 < 100K

---

<div align="center">

# IndEgo：面向第一人称助手的工业场景与协作工作数据集

**[Vivek Chavan](https://vivekchavan.com/)¹²*，[Yasmina Imgrund](https://www.linkedin.com/in/yasmina-imgrund/)²†，[Tung Dao](https://www.linkedin.com/in/lam-dao-tung/)²†，[Sanwantri Bai](https://www.linkedin.com/in/sanwantri-bai-0a808a1b3/)³†，[Bosong Wang](https://www.linkedin.com/in/bosong0106/)⁴†，Ze Lu⁵†，[Oliver Heimann](https://www.linkedin.com/in/oliver-heimann/)¹，[Jörg Krüger](https://www.tu.berlin/iat/ueber-uns/leitung)¹²**

<p>
¹弗劳恩霍夫IPK研究所，柏林 &nbsp;&nbsp; ²柏林工业大学 &nbsp;&nbsp; ³蒂宾根大学<br>
⁴亚琛工业大学 &nbsp;&nbsp; ⁵汉诺威莱布尼茨大学
</p>

*^{*项目负责人 &nbsp;&nbsp;&nbsp; †柏林弗劳恩霍夫IPK研究所学生毕业设计/项目期间完成的工作。}

<div align="center">
<h3 style="display: flex; align-items: center; justify-content: center; gap: 10px; margin-top: 1em; margin-bottom: 1em;">
<img src="https://IndEgo-Dataset.github.io/assets/NeurIPS-logo.svg" alt="NeurIPS标志" height="200">
<span>收录于 NeurIPS 2025</span>
</h3>
</div>

<p>
<a href="https://IndEgo-Dataset.github.io/" target="_blank"><img src="https://img.shields.io/badge/Project-主页-blue?style=flat-square" alt="项目主页"></a>
<a href="https://openreview.net/forum?id=jKw3Qhc8m1" target="_blank"><img src="https://img.shields.io/badge/Paper-OpenReview-red?style=flat-square" alt="论文PDF"></a>
<a href="https://github.com/Vivek9Chavan/IndEgo/" target="_blank"><img src="https://img.shields.io/badge/Code-GitHub-black?style=flat-square&logo=github" alt="代码仓库"></a>
<a href="https://neurips.cc/virtual/2025/poster/121501" target="_blank"><img src="https://img.shields.io/badge/NeurIPS-会议页面-orange?style=flat-square" alt="NeurIPS页面"></a>
</p>
<p>
<a href="https://colab.research.google.com/drive/1qCZnFQNRjBuy3vBlkMy7sMTcYkTNOzgg?usp=sharing" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="在Colab中打开"></a>
</p>

</div>

---

> [!WARNING]
> # 🚧 数据更新进行中 🚧
>
> ### ⚠️ 基于社区反馈，本数据集的结构正在进行重组。
> **文件路径与文件夹名称即将变更。**
>
> 若您当前下载数据集，本地文件结构将与未来更新后的版本不一致。我们建议您等待结构重组完成（预计完成时间：2025年12月12日）。
>
> **[👉 点击此处订阅数据集就绪通知](https://forms.gle/j8krD3At2hEWjJPg8)**

## 📖 摘要
我们提出**IndEgo**——一个多模态**第一人称视角（egocentric）与第三人称视角（exocentric）**视频数据集，涵盖装配/拆解、物流与组织、检测与维修以及木工等常见工业任务。本数据集包含**3460段第一视角录制视频（总时长约197小时）**与**1092段第三视角录制视频（总时长约97小时）**。


IndEgo的核心聚焦于**协作工作场景**，即两名工人协作完成兼具认知与体力挑战的任务。第一视角录制内容包含丰富的多模态数据：眼动追踪数据、口述文本、音频、运动数据以及半稠密点云。

我们提供：
- 精细标注：动作、摘要、错误标签与口述文本
- 处理后输出：眼动数据、手部姿态、基于同步定位与地图构建（Simultaneous Localization and Mapping, SLAM）的半稠密点云
- 基准测试任务：流程化/非流程化任务理解、**协作任务**、**错误检测**以及**基于推理的视频问答（Video QA）**

基准评估结果表明，IndEgo对当前顶尖的多模态模型构成了显著挑战。

---

## 🧩 引用格式
若您在研究工作中使用**IndEgo**数据集，请引用以下NeurIPS 2025论文：

bibtex
@inproceedings{Chavan2025IndEgo,
author = {Vivek Chavan and Yasmina Imgrund and Tung Dao and Sanwantri Bai and Bosong Wang and Ze Lu and Oliver Heimann and J{"o}rg Kr{"u}ger},
title = {IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track},
year = {2025},
url = {https://neurips.cc/virtual/2025/poster/121501}
}

## 致谢与资助
本工作得到德国联邦研究、技术与空间部（BMFTR）以及德国航空航天中心（DLR）在KIKERP项目（资助编号：16IS23055C）下的支持，该项目隶属于KI4KMU计划。我们感谢Meta AI与Reality Labs团队发起的Project Aria计划，包括其研究套件、配套工具与服务。同时感谢Hugging Face提供的公共数据集存储资助，使得IndEgo数据集能够大规模托管并向社区开放。数据采集工作在柏林工业大学机床与工厂管理研究所（IWF）的研究实验室与测试场完成。最后，我们衷心感谢所有参与数据采集的学生志愿者与工作人员。

<div style="display: flex; justify-content: center; align-items: center; flex-wrap: nowrap; gap: 25px; margin: 20px 0;">

<a href="https://www.bmftr.bund.de/" target="_blank">
<img src="https://raw.githubusercontent.com/IndEgo-Dataset/IndEgo-Dataset.github.io/main/assets/BMFTR_-_Logo_en.svg"
alt="德国联邦研究、技术与空间部" style="width:110px; height:auto;">
</a>

<a href="https://www.dlr.de/" target="_blank">
<img src="https://raw.githubusercontent.com/IndEgo-Dataset/IndEgo-Dataset.github.io/main/assets/PT_DLR_Logo_SW_D_2018_lang.png"
alt="德国航空航天中心" style="width:110px; height:auto;">
</a>

<a href="https://www.tu.berlin/iwf" target="_blank">
<img src="https://raw.githubusercontent.com/IndEgo-Dataset/IndEgo-Dataset.github.io/main/assets/logo-iwf-mit-namen-en.jpg"
alt="机床与工厂管理研究所" style="width:110px; height:auto;">
</a>

<a href="https://www.projectaria.com/" target="_blank"
style="font-weight:700; font-size:0.9rem; width:110px; text-align:center;">
Meta<br>现实实验室
</a>

<a href="https://huggingface.co/" target="_blank">
<img src="https://raw.githubusercontent.com/IndEgo-Dataset/IndEgo-Dataset.github.io/main/assets/Hf-logo-with-title.svg"
alt="Hugging Face" style="width:110px; height:auto;">
</a>

</div>