PLM-Video-Auto

# Dataset Card for PLM-Video Auto

[\[📃 Tech Report\]](https://arxiv.org/abs/2504.13180)
[\[📂 Github\]](https://github.com/facebookresearch/perception_models/)

Sythetic video captions and MCQs used in PLM, please refer to the paper, Section 3, for more details. The sythetic annotations covers: YT-1B, Ego4d with captions, YT-1B with MCQAs and Ego4d with QAs.

## Dataset Structure

### YT-1B Captions (yt1b_cap)

Data fields are :
- `video_id`: a `string` feature, unique identifier for the YouTube videoid.
- `scene_id`: a `string` feature, unique identifier for the `scene_id`.
- `start_time`: a `float` feature, to identify the start time of the scene.
- `end_time`: a `float` feature, to identify the end time of the scene.
- `data_engine_long_caption`: a `string` feature, long caption generated with PLM sythetic engine (Figure A9).
- `data_engine_short_caption`: a `string` feature, short caption generated with PLM sythetic engine (Figure A9).
- `plm_video_caption`: a `string` feature, PLM video caption generated by PLM (Figure A9).

A sample from YT-1B Captions looks as follows:
```
{
"video_id": "FnIDcoi9n2Y",
"scene_id": "2_0",
"start_time": 85.507334,
"end_time": 128.261,
"data_engine_long_caption": "A man stands in front of a banana tree talking to the camera and gesturing with his hands, the camera pans to a bunch of bananas hanging from the tree, the man continues talking and the camera shows the bananas again, a close-up of the bananas is shown with a hand reaching out to touch them, a phone number is displayed on a phone screen with a green tractor logo, the numbers are 080 6111 6333 and 9538510000.",
"data_engine_short_caption": "Man stands in front of banana tree, talks, gestures, bananas, phone number 080 6111 6333",
"plm_video_caption": "The video shows a man standing in front of a banana tree, talking to the camera. He gestures with his hands while speaking. The camera then pans to a bunch of bananas hanging from the tree. The man continues to talk to the camera, and the camera pans back to the bananas. The video ends with the camera showing the bananas again.",
}
```

### YT-1B MCQAs (yt1b_mcqa)

Data fields are:
- `video_id`: a `string` feature, unique identifier for the YouTube videoid.
- `start_time`: a `float` feature, to identify the start time of the scene.
- `end_time`: a `float` feature, to identify the end time of the scene.
- `conversations`: a `list` of `dict` feature, with the actual MCQA used to train PLM (Figure A9).
- `category`: a `string` feature to represent different MCQ types.

A sample from YT-1B Captions looks as follows:
```
{
"conversations": [
{
"from": "human",
"value": "\nWhat is written on the black screen at the end of the video?\nOptions:\n(A) Sports Telemundo\n(B) Telemundo Deportes\n(C) Deportes Telemundo\n(D) Deportes TV\n(E) Telemundo Sports\nAnswer with a letter.",
},
{"from": "gpt", "value": "(C) Deportes Telemundo"},
],
"data_type": "video_blob",
"category": "OCR Problems",
"video_id": "vGUEKYuibt8",
"start_time": 90.99,
"end_time": 94.861,
}
```

### Ego4d Caps (ego4d_cap)
Data fields are:
- `video_id`: a `string` feature, unique identifier for the Ego4D videoid.
- `start_time`: a `float` feature, to identify the start time in sec of the scene.
- `end_time`: a `float` feature, to identify the end time in sec of the scene.
- `conversations`: a `list` of `dict` feature, with the actual prompt and caption used to train PLM.

A sample from Ego4d Captions looks as follows:
```
{
"video_id": "36ee5431-8d3f-4773-b158-2b42932666aa",
"start_time": 120.0,
"end_time": 180.0,
"conversations": [
{
"from": "human",
"value": "\nWhat is happening in the video, and what actions is the camera wearer C performing?",
},
{
"from": "gpt",
"value": "The camera wearer changes the setting of a video game on a computer monitor sitting on top of a desk next to a chair, with various objects such as wires, books, and other items scattered on the desk and a wall in the background. They then play the video game using a mouse, with the keyboard and other gaming peripherals visible on the desk. After a while, they remove adds from their desktop, with papers, books, cables, and other objects still cluttering the desk. The camera wearer then continues playing the video game with the mouse, actively engaging in the multiplayer online battle arena game displayed on the computer screen, using the keyboard and mouse to control their character and navigate the fantasy game environment.",
},
],
}
```

### Ego4d QA (ego4d_qa)
Data fields are:
- `video_id`: a `string` feature, unique identifier for the Ego4D videoid.
- `start_time`: a `float` feature, to identify the start time in sec of the scene.
- `end_time`: a `float` feature, to identify the end time in sec of the scene.
- `conversations`: a `list` of `dict` feature, with the actual question and answer used to train PLM.

A sample from Ego4D QA looks as follows:
```
{
"video_id": "8d928865-5d5f-4b10-b1cb-ef439c5c8ecd",
"start_time": 60.0,
"end_time": 120.0,
"conversations": [
{
"from": "human",
"value": "\nWhat objects are on the table in front of the camera wearer?",
},
{"from": "gpt", "value": "There are various objects, and a sewing machine."},
],
}
```

### Data Stats

| | Sample | Type |
| ----------- | ----------- | ----------- |
| yt1b_cap | 12134877 | Captions |
| yt1b_mcqa | 3383670 | MCQAs |
| ego4d_cap | 183029 | Captions |
| ego4d_qa | 703935 | QAs |

### Licensing Information

This data is an output from Llama 3.2, and subject to the Llama 3.2 license (https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE).
Use of the data to train, fine tune, or otherwise improve an AI model, which is distributed or made available, shall also include "Llama" at the beginning of any such AI model name.

### Citation Information

Cite as:
```
@article{cho2025PerceptionLM,
title={PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding},
author={Jang Hyun Cho and Andrea Madotto and Effrosyni Mavroudi and Triantafyllos Afouras and Tushar Nagarajan and Muhammad Maaz and Yale Song and Tengyu Ma and Shuming Hu and Hanoona Rasheed and Peize Sun and Po-Yao Huang and Daniel Bolya and Suyog Jain and Miguel Martin and Huiyu Wang and Nikhila Ravi and Shashank Jain and Temmy Stark and Shane Moon and Babak Damavandi and Vivian Lee and Andrew Westbury and Salman Khan and Philipp Kr\"{a}henb\"{u}hl and Piotr Doll{\'a}r and Lorenzo Torresani and Kristen Grauman and Christoph Feichtenhofer},
journal={arXiv},
year={2025}
}
```

# PLM-Video Auto 数据集卡片
[📃 技术报告](https://arxiv.org/abs/2504.13180)
[📢 Github仓库](https://github.com/facebookresearch/perception_models/)

本数据集包含用于感知语言模型（PerceptionLM，PLM）的合成视频字幕与多项选择题问答（Multiple Choice Question Answering，MCQA）样本，详细信息请参阅论文第3节。本次合成标注覆盖四类数据：带字幕的YT-1B、带字幕的Ego4D、带MCQA的YT-1B以及带问答的Ego4D。

## 数据集结构

### YT-1B 字幕数据集（yt1b_cap）
数据字段如下：
- `video_id`：字符串类型特征，为YouTube视频的唯一标识符。
- `scene_id`：字符串类型特征，为场景的唯一标识符。
- `start_time`：浮点型特征，用于标识场景的起始时间。
- `end_time`：浮点型特征，用于标识场景的结束时间。
- `data_engine_long_caption`：字符串类型特征，由PLM合成引擎生成的长字幕（见图A9）。
- `data_engine_short_caption`：字符串类型特征，由PLM合成引擎生成的短字幕（见图A9）。
- `plm_video_caption`：字符串类型特征，由PLM生成的视频字幕（见图A9）。

YT-1B字幕数据集的样本示例如下：

{
"video_id": "FnIDcoi9n2Y",
"scene_id": "2_0",
"start_time": 85.507334,
"end_time": 128.261,
"data_engine_long_caption": "A man stands in front of a banana tree talking to the camera and gesturing with his hands, the camera pans to a bunch of bananas hanging from the tree, the man continues talking and the camera shows the bananas again, a close-up of the bananas is shown with a hand reaching out to touch them, a phone number is displayed on a phone screen with a green tractor logo, the numbers are 080 6111 6333 and 9538510000.",
"data_engine_short_caption": "Man stands in front of banana tree, talks, gestures, bananas, phone number 080 6111 6333",
"plm_video_caption": "The video shows a man standing in front of a banana tree, talking to the camera. He gestures with his hands while speaking. The camera then pans to a bunch of bananas hanging from the tree. The man continues to talk to the camera, and the camera pans back to the bananas. The video ends with the camera showing the bananas again.",
}

### YT-1B 多项选择题问答数据集（yt1b_mcqa）
数据字段如下：
- `video_id`：字符串类型特征，为YouTube视频的唯一标识符。
- `start_time`：浮点型特征，用于标识场景的起始时间。
- `end_time`：浮点型特征，用于标识场景的结束时间。
- `conversations`：字典列表类型特征，包含用于训练PLM的实际MCQA样本（见图A9）。
- `category`：字符串类型特征，用于表示不同的MCQ题型。

YT-1B多项选择题问答数据集的样本示例如下：

{
"conversations": [
{
"from": "human",
"value": "
What is written on the black screen at the end of the video?
Options:
(A) Sports Telemundo
(B) Telemundo Deportes
(C) Deportes Telemundo
(D) Deportes TV
(E) Telemundo Sports
Answer with a letter.",
},
{"from": "gpt", "value": "(C) Deportes Telemundo"},
],
"data_type": "video_blob",
"category": "OCR Problems",
"video_id": "vGUEKYuibt8",
"start_time": 90.99,
"end_time": 94.861,
}

### Ego4D 字幕数据集（ego4d_cap）
数据字段如下：
- `video_id`：字符串类型特征，为Ego4D视频的唯一标识符。
- `start_time`：浮点型特征，用于标识场景的起始时间（单位：秒）。
- `end_time`：浮点型特征，用于标识场景的结束时间（单位：秒）。
- `conversations`：字典列表类型特征，包含用于训练PLM的实际提示词与字幕。

Ego4D字幕数据集的样本示例如下：

{
"video_id": "36ee5431-8d3f-4773-b158-2b42932666aa",
"start_time": 120.0,
"end_time": 180.0,
"conversations": [
{
"from": "human",
"value": "
What is happening in the video, and what actions is the camera wearer C performing?",
},
{
"from": "gpt",
"value": "The camera wearer changes the setting of a video game on a computer monitor sitting on top of a desk next to a chair, with various objects such as wires, books, and other items scattered on the desk and a wall in the background. They then play the video game using a mouse, with the keyboard and other gaming peripherals visible on the desk. After a while, they remove adds from their desktop, with papers, books, cables, and other objects still cluttering the desk. The camera wearer then continues playing the video game with the mouse, actively engaging in the multiplayer online battle arena game displayed on the computer screen, using the keyboard and mouse to control their character and navigate the fantasy game environment.",
},
],
}

### Ego4D 问答数据集（ego4d_qa）
数据字段如下：
- `video_id`：字符串类型特征，为Ego4D视频的唯一标识符。
- `start_time`：浮点型特征，用于标识场景的起始时间（单位：秒）。
- `end_time`：浮点型特征，用于标识场景的结束时间（单位：秒）。
- `conversations`：字典列表类型特征，包含用于训练PLM的实际问题与答案。

Ego4D问答数据集的样本示例如下：

{
"video_id": "8d928865-5d5f-4b10-b1cb-ef439c5c8ecd",
"start_time": 60.0,
"end_time": 120.0,
"conversations": [
{
"from": "human",
"value": "
What objects are on the table in front of the camera wearer?",
},
{"from": "gpt", "value": "There are various objects, and a sewing machine."},
],
}

### 数据统计
| | 样本数量 | 数据类型 |
| ----------- | ----------- | ----------- |
| yt1b_cap | 12134877 | 字幕数据集 |
| yt1b_mcqa | 3383670 | 多项选择题问答数据集 |
| ego4d_cap | 183029 | 字幕数据集 |
| ego4d_qa | 703935 | 问答数据集 |

### 许可信息
本数据集为Llama 3.2的输出产物，受Llama 3.2许可协议约束（https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE）。若使用本数据集训练、微调或以其他方式改进人工智能模型并进行分发或公开提供，则该人工智能模型的名称需以“Llama”作为前缀。

### 引用信息
引用格式如下：

@article{cho2025PerceptionLM,
title={PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding},
author={Jang Hyun Cho and Andrea Madotto and Effrosyni Mavroudi and Triantafyllos Afouras and Tushar Nagarajan and Muhammad Maaz and Yale Song and Tengyu Ma and Shuming Hu and Hanoona Rasheed and Peize Sun and Po-Yao Huang and Daniel Bolya and Suyog Jain and Miguel Martin and Huiyu Wang and Nikhila Ravi and Shashank Jain and Temmy Stark and Shane Moon and Babak Damavandi and Vivian Lee and Andrew Westbury and Salman Khan and Philipp Krähenbühl and Piotr Dollár and Lorenzo Torresani and Kristen Grauman and Christoph Feichtenhofer},
journal={arXiv},
year={2025}
}

搜集汇总

背景与挑战

背景概述

PLM-Video-Auto是由Facebook发布的合成视频标注数据集，包含来自YT-1B和Ego4d的约1600万条样本，涵盖视频描述、多项选择题和问答对，用于训练PLM模型。该数据集大小为8.06GB，采用Apache License 2.0许可。