action100m-preview

# Action100M: A Large-scale Video Action Dataset

[**Paper**](https://huggingface.co/papers/2601.10592) | [**GitHub**](https://github.com/facebookresearch/Action100M)

Action100M is a large-scale dataset constructed from 1.2M Internet instructional videos (14.6 years of duration), yielding ~100 million temporally localized segments with open-vocabulary action supervision and rich captions. It serves as a foundation for scalable research in video understanding and world modeling.

## Load Action100M Annotations

Our data can be loaded from the 🤗 huggingface repo at [`facebook/action100m-preview`](https://huggingface.co/datasets/facebook/action100m-preview) where we released 10% of the full Action100M for preview. For examples of loading from local parquet files (from cloned repo) and visualization, see the [GitHub repo](https://github.com/facebookresearch/action100m).

```python
from datasets import load_dataset

dataset = load_dataset(
"parquet",
data_files=f"hf://datasets/facebook/Action100M-preview/data/*.parquet",
streaming=True,
)
it = iter(dataset["train"])

sample = next(it)
```

Each `sample` loaded above contains all annotations for one video, and it has three fields:

* `video_uid` *(string)*: YouTube video id of the source video.
* `metadata` *(dict)*: video-level metadata (title / description / ASR transcript, etc.)
* `nodes` *(list[dict])*: annotations for each segments.

Each element in `nodes` is a temporally localized segment in the hierachical Tree-of-Captions, it contains:

* `start`, `end` *(float)*: segment boundaries in seconds within the full video.
* `node_id` *(string)*: unique id of this segment node.
* `parent_id` *(string or null)*: id of the parent segment. The root node (corresponding to the entire video) has `parent_id = null`.
* `level` *(int)*: depth in the hierarchy. Smaller `level` is coarser (longer segments); larger `level` is finer (shorter segments).
* `plm_caption` *(string or null)*: a caption generated by PLM-3B for this segment.
* `plm_action` *(string or null)*: a short action label produced by PLM-3B.
* `llama3_caption` *(string or null)*: middle frame caption produced by LLama-3.2-Vision-11B for leaf nodes.
* `gpt` *(dict or null)*: main Action100M annotations, available for segments that is not too short:

* `gpt["summary"]["brief"]`: one-sentence concise caption of the segment.
* `gpt["summary"]["detailed"]`: longer, detailed summarization of the video segment.
* `gpt["action"]["brief"]`: short verb phrase naming the step.
* `gpt["action"]["detailed"]`: imperative-style instruction describing how the action is done.
* `gpt["action"]["actor"]`: who/what performs the action (noun phrase).

## Citation

```bibtex
@article{chen2026action100m,
title={Action100M: A Large-scale Video Action Dataset},
author={Chen, Delong and Kasarla, Tejaswi and Bang, Yejin and Shukor, Mustafa and Chung, Willy and Yu, Jade and Bolourchi, Allen and Moutakanni, Théo and Fung, Pascale},
journal={arXiv preprint arXiv:2601.10592},
year={2026}
}
```