FraunhoferIPK/IndEgo

---
license: cc-by-4.0
task_categories:
- visual-question-answering
- summarization
- video-classification
- any-to-any
language:
- en
- de
pretty_name: IndEgo
tags:
- industrial
- egocentric
- procedural
- collaborative work
- mistake detection
- VQA
- video understanding
size_categories:
- 10K<n<100K
---

<div align="center">

# IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants

**[Vivek Chavan](https://vivekchavan.com/)¹²\*, [Yasmina Imgrund](https://www.linkedin.com/in/yasmina-imgrund/)²†, [Tung Dao](https://www.linkedin.com/in/lam-dao-tung/)²†, [Sanwantri Bai](https://www.linkedin.com/in/sanwantri-bai-0a808a1b3/)³†, [Bosong Wang](https://www.linkedin.com/in/bosong0106/)⁴†, Ze Lu⁵†, [Oliver Heimann](https://www.linkedin.com/in/oliver-heimann/)¹, [Jörg Krüger](https://www.tu.berlin/iat/ueber-uns/leitung)¹²**

<p>
¹Fraunhofer IPK, Berlin &nbsp;&nbsp; ²Technical University of Berlin &nbsp;&nbsp; ³University of Tübingen<br>
⁴RWTH Aachen University &nbsp;&nbsp; ⁵Leibniz University Hannover
</p>

*^{\*Project Lead &nbsp;&nbsp;&nbsp; †Work done during student theses/projects at Fraunhofer IPK, Berlin.}*

<div align="center">
<h3 style="display: flex; align-items: center; justify-content: center; gap: 10px; margin-top: 1em; margin-bottom: 1em;">
<img src="https://IndEgo-Dataset.github.io/assets/NeurIPS-logo.svg" alt="NeurIPS Logo" height="200">
<span>Published at NeurIPS 2025</span>
</h3>
</div>

<p>
<a href="https://IndEgo-Dataset.github.io/" target="_blank"><img src="https://img.shields.io/badge/Project-Website-blue?style=flat-square" alt="Project Website"></a>
<a href="https://openreview.net/forum?id=jKw3Qhc8m1" target="_blank"><img src="https://img.shields.io/badge/Paper-OpenReview-red?style=flat-square" alt="Paper PDF"></a>
<a href="https://github.com/Vivek9Chavan/IndEgo/" target="_blank"><img src="https://img.shields.io/badge/Code-GitHub-black?style=flat-square&logo=github" alt="Code"></a>
<a href="https://neurips.cc/virtual/2025/poster/121501" target="_blank"><img src="https://img.shields.io/badge/NeurIPS-Page-orange?style=flat-square" alt="NeurIPS Page"></a>
</p>
<p>
<a href="https://colab.research.google.com/drive/1qCZnFQNRjBuy3vBlkMy7sMTcYkTNOzgg?usp=sharing" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
</p>

</div>

---

> [!WARNING]
> # 🚧 UPDATE IN PROGRESS 🚧
>
> ### ⚠️ Based on the feedback from other community members, the dataset structure is being reorganised.
> **File paths and folder names are changing.**
>
> If you download the data right now, your local file structure may become inconsistent with future updates.
> We recommend waiting until the restructuring is complete (ETA: 12 Dec, 2025).
>
> **[👉 Click here to be notified when the dataset is ready](https://forms.gle/j8krD3At2hEWjJPg8)**

## 📖 Abstract

We introduce **IndEgo**, a multimodal **egocentric and exocentric** video dataset capturing common industrial tasks such as assembly/disassembly, logistics and organisation, inspection and repair, and woodworking. The dataset includes **3,460 egocentric recordings (~197 hours)** and **1,092 exocentric recordings (~97 hours)**.


A central focus of IndEgo is **collaborative work**, where two workers coordinate on cognitively and physically demanding tasks. The egocentric recordings include rich multimodal data — eye gaze, narration, sound, motion, and semi-dense point clouds.

We provide:
- Detailed annotations: actions, summaries, mistake labels, and narrations
- Processed outputs: eye gaze, hand poses, SLAM-based semi-dense point clouds
- Benchmarks: procedural/non-procedural task understanding, **collaborative tasks**, **Mistake Detection**, and **reasoning-based Video QA**

Baseline evaluations show that IndEgo presents a challenge for state-of-the-art multimodal models.

---

## 🧩 Citation

If you use **IndEgo** in your research, please cite our NeurIPS 2025 paper:

```bibtex
@inproceedings{Chavan2025IndEgo,
author = {Vivek Chavan and Yasmina Imgrund and Tung Dao and Sanwantri Bai and Bosong Wang and Ze Lu and Oliver Heimann and J{\"o}rg Kr{\"u}ger},
title = {IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track},
year = {2025},
url = {https://neurips.cc/virtual/2025/poster/121501}
}
```

## Acknowledgments & Funding

This work is supported by the German Federal Ministry of Research, Technology and Space (BMFTR) and the German Aerospace Center (DLR) under the KIKERP project (Grant No. 16IS23055C) within the KI4KMU program. We are grateful to the Meta AI and Reality Labs teams for the Project Aria initiative, including the research kit, associated tools, and services. We also thank Hugging Face for providing a public-dataset storage grant that enables large-scale hosting and community access to the IndEgo dataset. Data collection was conducted at the research labs and test field of the Institute of Machine Tools and Factory Management (IWF), TU Berlin. Finally, we extend our sincere thanks to all student volunteers and workers who contributed to the data collection.

<div style="display: flex; justify-content: center; align-items: center; flex-wrap: nowrap; gap: 25px; margin: 20px 0;">

<a href="https://www.bmftr.bund.de/" target="_blank">
<img src="https://raw.githubusercontent.com/IndEgo-Dataset/IndEgo-Dataset.github.io/main/assets/BMFTR_-_Logo_en.svg"
alt="BMBF" style="width:110px; height:auto;">
</a>

<a href="https://www.dlr.de/" target="_blank">
<img src="https://raw.githubusercontent.com/IndEgo-Dataset/IndEgo-Dataset.github.io/main/assets/PT_DLR_Logo_SW_D_2018_lang.png"
alt="DLR" style="width:110px; height:auto;">
</a>

<a href="https://www.tu.berlin/iwf" target="_blank">
<img src="https://raw.githubusercontent.com/IndEgo-Dataset/IndEgo-Dataset.github.io/main/assets/logo-iwf-mit-namen-en.jpg"
alt="IWF" style="width:110px; height:auto;">
</a>

<a href="https://www.projectaria.com/" target="_blank"
style="font-weight:700; font-size:0.9rem; width:110px; text-align:center;">
Meta<br>Reality Labs
</a>

<a href="https://huggingface.co/" target="_blank">
<img src="https://raw.githubusercontent.com/IndEgo-Dataset/IndEgo-Dataset.github.io/main/assets/Hf-logo-with-title.svg"
alt="Hugging Face" style="width:110px; height:auto;">
</a>

</div>