tomg-group-umd/cinepile

---
language:
- en
license: cc-by-nc-sa-4.0
size_categories:
- 100K<n<1M
task_categories:
- visual-question-answering
- video-text-to-text
dataset_info:
- config_name: default
features:
- name: movie_name
dtype: string
- name: year
dtype: int64
- name: genre
sequence: string
- name: yt_clip_title
dtype: string
- name: yt_clip_link
dtype: string
- name: movie_scene
dtype: string
- name: subtitles
dtype: string
- name: question
dtype: string
- name: choices
sequence: string
- name: answer_key
dtype: string
- name: answer_key_position
dtype: int64
- name: question_category
dtype: string
- name: hard_split
dtype: string
- name: visual_reliance
dtype: string
splits:
- name: train
num_bytes: 1207285134
num_examples: 298888
- name: test
num_bytes: 18238920
num_examples: 4941
download_size: 58053414
dataset_size: 1225524054
- config_name: v2
default: True
features:
- name: movie_name
dtype: string
- name: year
dtype: int64
- name: genre
sequence: string
- name: yt_clip_title
dtype: string
- name: yt_clip_link
dtype: string
- name: movie_scene
dtype: string
- name: subtitles
dtype: string
- name: question
dtype: string
- name: choices
sequence: string
- name: answer_key
dtype: string
- name: answer_key_position
dtype: int64
- name: question_category
dtype: string
- name: hard_split
dtype: string
- name: visual_reliance
dtype: string
- name: videoID
dtype: string
splits:
- name: train
num_bytes: 1226448710
num_examples: 298888
- name: test
num_bytes: 18430889
num_examples: 4941
download_size: 69504867
dataset_size: 1244879599
configs:
- config_name: v1
data_files:
- split: train
path: v1/train-*
- split: test
path: v1/test-*
- config_name: v2
default: True
data_files:
- split: train
path: v2/train-*
- split: test
path: v2/test-*
extra_gated_prompt: 'The CinePile dataset provides links to YouTube videos as part
of its data collection. CinePile does not own any of the content linked within this
dataset. Ownership and copyright of the videos belong to the respective YouTube
channel owners. It is the responsibility of these source channels to ensure that
all content follows the terms and conditions set by YouTube. By accessing this dataset,
you acknowledge and agree that:'
extra_gated_fields:
I understand that CinePile does not own the YouTube videos linked in this dataset: checkbox
I agree to use this dataset for non-commercial use ONLY: checkbox
I agree with the data license for this dataset: checkbox
---

# CinePile: A Long Video Question Answering Dataset and Benchmark

CinePile is a question-answering-based, long-form video understanding dataset. It has been created using advanced large language models (LLMs) with human-in-the-loop pipeline leveraging existing human-generated raw data. It consists of approximately 300,000 training data points and 5,000 test data points.

If you have any comments or questions, reach out to: [Ruchit Rawal](https://ruchitrawal.github.io/) or [Gowthami Somepalli](https://somepago.github.io/)

Other links - [Website](https://ruchitrawal.github.io/cinepile/) &ensp; [Paper](https://arxiv.org/abs/2405.08813)

## Version support and revisions

- October 2024: We refine both the training and test split using the adversarial refinement process described in detail [here](https://huggingface.co/blog/cinepile2). This refined version will be loaded by default when running `load_dataset("tomg-group-umd/cinepile")`. To load the previous version, use `load_dataset("tomg-group-umd/cinepile", "v1")`.

## Dataset Structure

Each row in the dataset consists of a `question` (dtype: string), five `choices` (dtype: list), and an `answer_key` (dtype: string). Auxiliary columns are included that store the movie's name, movie's genre, video clip titles, etc.

The train split of the dataset is intended for the instruction tuning of video-LLMs. The test split is designed for benchmarking video-LLMs and includes the `hard_split` column, which is "True" for particularly challenging questions and "False" otherwise. The `visual_reliance` column indicates whether a question likely requires integrating visual information to be answered correctly.

### Dataset Features

- **movie_name**: Name of the movie to which the video clip belongs.
- **year**: Release year of the movie.
- **genre**: Genre(s) of the movie.
- **yt_clip_title**: Title of the video clip as it appears on YouTube.
- **yt_clip_link**: URL link to the video clip on YouTube.
- **movie_scene**: Description of the movie scene, contains subtitles and visual descriptions.
- **subtitles**: Subtitles extracted from the movie scene.
- **question**: Question derived from the movie scene.
- **choices**: Multiple-choice options associated with the question.
- **answer_key**: The correct answer from the choices provided.
- **answer_key_position**: The index position of the correct answer within the choices list.
- **question_category**: The category to which the question belongs.
- **hard_split**: Indicates if the question is particularly challenging. "N/A" for the train set; applicable only in the test set.
- **visual_reliance**: Indicates if the question requires visual information for an accurate answer. "N/A" for the train set.

## Dataset Use and Starter Snippets

### Loading the dataset
You can load the dataset easily using the Datasets library:
```
from datasets import load_dataset

dataset = load_dataset("tomg-group-umd/cinepile")
```

### Retrieving questions from a specific clip
```
cinepile_test = load_dataset('tomg-group-umd/cinepile', token=True, split='test')
yt_clip_title = "Extraction (2015) - You're Crazy Scene (5/10) | Movieclips"

clip_test_dataset = cinepile_test.filter(lambda x: x['yt_clip_title'] == yt_clip_title)
```

### Loading the hard-split:
```
cinepile_test = load_dataset('tomg-group-umd/cinepile', token=True, split='test')
hard_split_test = cinepile_test.filter(lambda x: x['hard_split'] == "True")
```

Please refer to the accompanying [Colab notebook](https://colab.research.google.com/drive/1jDwvPoCsg9tck3dFhVCV-h3Ny6992wCr?usp=sharing) for more examples e.g. evaluating VLMs, extracting responses, etc.

### Cite us:

```
@article{rawal2024cinepile,
title={CinePile: A Long Video Question Answering Dataset and Benchmark},
author={Rawal, Ruchit and Saifullah, Khalid and Basri, Ronen and Jacobs, David and Somepalli, Gowthami and Goldstein, Tom},
journal={arXiv preprint arXiv:2405.08813},
year={2024}
}
```

语言：
- 英语
许可协议：CC BY-NC-SA 4.0
样本规模区间：10万 < 样本数 < 100万
任务类别：
- 视觉问答（Visual Question Answering）
- 视频到文本生成（Video-Text-to-Text）
数据集信息：
- 配置名称：default
特征：
- 名称：movie_name，数据类型：字符串
- 名称：year，数据类型：整数
- 名称：genre，数据类型：字符串序列
- 名称：yt_clip_title，数据类型：字符串
- 名称：yt_clip_link，数据类型：字符串
- 名称：movie_scene，数据类型：字符串
- 名称：subtitles，数据类型：字符串
- 名称：question，数据类型：字符串
- 名称：choices，数据类型：字符串序列
- 名称：answer_key，数据类型：字符串
- 名称：answer_key_position，数据类型：整数
- 名称：question_category，数据类型：字符串
- 名称：hard_split，数据类型：字符串
- 名称：visual_reliance，数据类型：字符串
划分：
- 名称：train（训练集），字节数：1207285134，样本数：298888
- 名称：test（测试集），字节数：18238920，样本数：4941
下载大小：58053414
数据集总大小：1225524054
- 配置名称：v2，默认配置：是
特征：
- 名称：movie_name，数据类型：字符串
- 名称：year，数据类型：整数
- 名称：genre，数据类型：字符串序列
- 名称：yt_clip_title，数据类型：字符串
- 名称：yt_clip_link，数据类型：字符串
- 名称：movie_scene，数据类型：字符串
- 名称：subtitles，数据类型：字符串
- 名称：question，数据类型：字符串
- 名称：choices，数据类型：字符串序列
- 名称：answer_key，数据类型：字符串
- 名称：answer_key_position，数据类型：整数
- 名称：question_category，数据类型：字符串
- 名称：hard_split，数据类型：字符串
- 名称：visual_reliance，数据类型：字符串
- 名称：videoID，数据类型：字符串
划分：
- 名称：train（训练集），字节数：1226448710，样本数：298888
- 名称：test（测试集），字节数：18430889，样本数：4941
下载大小：69504867
数据集总大小：1244879599
配置项：
- 配置名称：v1
数据文件：
- 划分：train（训练集），路径：v1/train-*
- 划分：test（测试集），路径：v1/test-*
- 配置名称：v2，默认配置：是
数据文件：
- 划分：train（训练集），路径：v2/train-*
- 划分：test（测试集），路径：v2/test-*
额外申请提示：CinePile数据集在数据收集中包含了YouTube视频的链接。CinePile并不拥有本数据集中所链接的任何内容，视频的所有权与版权归对应YouTube频道的所有者所有。这些来源频道需确保其所有内容符合YouTube设定的条款与条件。访问本数据集即表示您知晓并同意以下条款：
额外申请字段：
1. 我知晓CinePile并不拥有本数据集中所链接的YouTube视频：复选框
2. 我同意仅将本数据集用于非商业用途：复选框
3. 我同意本数据集的使用许可协议：复选框

# CinePile：一款长视频问答数据集与基准测试集

CinePile是一款基于问答范式的长视频理解数据集，其构建依托先进的大语言模型（Large Language Model, LLM），并采用人机协同的闭环流水线，利用已有的人工生成原始数据完成。该数据集包含约30万条训练样本与5000条测试样本。

若您有任何意见或疑问，请联系：[Ruchit Rawal](https://ruchitrawal.github.io/) 或 [Gowthami Somepalli](https://somepago.github.io/)
其他相关链接：[官方网站](https://ruchitrawal.github.io/cinepile/) 与 [论文](https://arxiv.org/abs/2405.08813)

## 版本支持与修订
- 2024年10月：我们采用详细阐述于[此处](https://huggingface.co/blog/cinepile2)的对抗性优化流程，对训练集与测试集均进行了优化。运行`load_dataset("tomg-group-umd/cinepile")`时将默认加载此优化后的版本。若需加载旧版本，请使用`load_dataset("tomg-group-umd/cinepile", "v1")`。

## 数据集结构
数据集的每一行均包含一个`question`（数据类型：字符串）、五个`choices`（数据类型：列表）以及一个`answer_key`（数据类型：字符串）。此外还包含辅助列，用于存储电影名称、电影类型、视频片段标题等信息。

训练集用于视频大语言模型（Video-LLM）的指令微调，测试集则用于视频大语言模型的基准测试，其中包含`hard_split`列：对于极具挑战性的问题，该字段值为“True”，否则为“False”。`visual_reliance`列用于标识某一问题是否需要整合视觉信息才能正确作答。

### 数据集特征
- **movie_name**：视频片段所属电影的名称。
- **year**：电影的上映年份。
- **genre**：电影的类型（可包含多个类型）。
- **yt_clip_title**：YouTube平台上展示的视频片段标题。
- **yt_clip_link**：YouTube平台上该视频片段的URL链接。
- **movie_scene**：电影场景的描述，包含字幕与视觉描述信息。
- **subtitles**：从电影场景中提取的字幕内容。
- **question**：基于电影场景生成的问题。
- **choices**：与该问题关联的多项选择选项。
- **answer_key**：给定选项中的正确答案。
- **answer_key_position**：正确答案在选项列表中的索引位置。
- **question_category**：问题所属的类别。
- **hard_split**：标识该问题是否极具挑战性。训练集该字段为“N/A”，仅在测试集中生效。
- **visual_reliance**：标识该问题是否需要视觉信息才能得到准确答案。训练集该字段为“N/A”。

## 数据集使用与入门代码示例
### 加载数据集
您可通过Hugging Face Datasets库轻松加载本数据集：

from datasets import load_dataset

dataset = load_dataset("tomg-group-umd/cinepile")

### 从指定视频片段中检索问题

cinepile_test = load_dataset('tomg-group-umd/cinepile', token=True, split='test')
yt_clip_title = "Extraction (2015) - You're Crazy Scene (5/10) | Movieclips"

clip_test_dataset = cinepile_test.filter(lambda x: x['yt_clip_title'] == yt_clip_title)

### 加载难度划分的测试集

cinepile_test = load_dataset('tomg-group-umd/cinepile', token=True, split='test')
hard_split_test = cinepile_test.filter(lambda x: x['hard_split'] == "True")

请参考配套的[Colab笔记本](https://colab.research.google.com/drive/1jDwvPoCsg9tck3dFhVCV-h3Ny6992wCr?usp=sharing)以获取更多示例，例如评估视觉语言模型（Vision-Language Model, VLM）、提取模型响应等。

### 引用我们

@article{rawal2024cinepile,
title={CinePile: A Long Video Question Answering Dataset and Benchmark},
author={Rawal, Ruchit and Saifullah, Khalid and Basri, Ronen and Jacobs, David and Somepalli, Gowthami and Goldstein, Tom},
journal={arXiv preprint arXiv:2405.08813},
year={2024}
}