VideoFeedback

[📃Paper](https://arxiv.org/abs/2406.15252) | [🌐Website](https://tiger-ai-lab.github.io/VideoScore/) | [💻Github](https://github.com/TIGER-AI-Lab/VideoScore) | [🛢️Datasets](https://huggingface.co/datasets/TIGER-Lab/VideoFeedback) | [🤗Model](https://huggingface.co/TIGER-Lab/VideoScore) | [🤗Demo](https://huggingface.co/spaces/TIGER-Lab/VideoScore)

## Overview
VideoFeedback contains a total of 37.6K text-to-video pairs from 11 popular video generative models,
with some real-world videos as data augmentation.
The videos are annotated by raters for five evaluation dimensions:
Visual Quality, Temporal Consistency, Dynamic Degree,
Text-to-Video Alignment and Factual Consistency, in 1-4 scoring scale.
VideoFeedback is used to for trainging of [VideoScore](https://huggingface.co/TIGER-Lab/VideoScore)

Below we show the detailed description of our VideoFeedback dataset.

## Load Dataset
- Load certain split and certain config of dataset
```python
from datasets import load_dataset
dataset=load_dataset("TIGER-Lab/Video_feedback",name="annotated",split="test")
print(dataset[0].keys())
print(dataset[0]["id"])
"""
dict_keys(['id', 'images', 'text prompt', 'video link', 'visual quality', 'temporal consistency', 'dynamic degree', 'text-to-video alignment', 'factual consistency', 'conversations'])
7001917
"""
```

- Load all subsets/configs of certain split
```python
from datasets import get_dataset_config_names, load_dataset
for config_name in get_dataset_config_names("TIGER-Lab/VideoFeedback"):
test_data = load_dataset("TIGER-Lab/VideoFeedback", name=config_name, split="test")
print(test_data[0]["id"])
"""
7001917
p006515
"""
```

## Citation
```bibtex
@article{he2024videoscore,
title = {VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation},
author = {He, Xuan and Jiang, Dongfu and Zhang, Ge and Ku, Max and Soni, Achint and Siu, Sherman and Chen, Haonan and Chandra, Abhranil and Jiang, Ziyan and Arulraj, Aaran and Wang, Kai and Do, Quy Duc and Ni, Yuansheng and Lyu, Bohan and Narsupalli, Yaswanth and Fan, Rongqi and Lyu, Zhiheng and Lin, Yuchen and Chen, Wenhu},
journal = {ArXiv},
year = {2024},
volume={abs/2406.15252},
url = {https://arxiv.org/abs/2406.15252},
}
```

[📃论文](https://arxiv.org/abs/2406.15252) | [🌐官网](https://tiger-ai-lab.github.io/VideoScore/) | [💻GitHub仓库](https://github.com/TIGER-AI-Lab/VideoScore) | [🛢️数据集](https://huggingface.co/datasets/TIGER-Lab/VideoFeedback) | [🤗模型](https://huggingface.co/TIGER-Lab/VideoScore) | [🤗演示Demo](https://huggingface.co/spaces/TIGER-Lab/VideoScore)

## 概述
VideoFeedback 数据集共包含来自11款主流视频生成模型的37.6万条文本-视频配对数据，并辅以部分真实世界视频以扩充数据集规模。所有视频均由标注人员从五大评估维度进行评分：视觉质量（Visual Quality）、时间一致性（Temporal Consistency）、动态程度（Dynamic Degree）、文本-视频对齐度（Text-to-Video Alignment）与事实一致性（Factual Consistency），评分区间为1至4分。VideoFeedback 数据集可用于训练[VideoScore](https://huggingface.co/TIGER-Lab/VideoScore)模型。下文将详细介绍VideoFeedback数据集。

## 加载数据集
### 加载指定划分与指定配置的数据集
python
from datasets import load_dataset
dataset=load_dataset("TIGER-Lab/Video_feedback",name="annotated",split="test")
print(dataset[0].keys())
print(dataset[0]["id"])
"""
字典键列表：['id', 'images', 'text prompt', 'video link', '视觉质量', '时间一致性', '动态程度', '文本-视频对齐度', '事实一致性', 'conversations']
7001917
"""

### 加载指定划分下的所有子集/配置
python
from datasets import get_dataset_config_names, load_dataset
for config_name in get_dataset_config_names("TIGER-Lab/VideoFeedback"):
test_data = load_dataset("TIGER-Lab/VideoFeedback", name=config_name, split="test")
print(test_data[0]["id"])
"""
7001917
p006515
"""

## 引用
bibtex
@article{he2024videoscore,
title = {VideoScore：构建用于模拟视频生成细粒度人类反馈的自动评估指标},
author = {He, Xuan and Jiang, Dongfu and Zhang, Ge and Ku, Max and Soni, Achint and Siu, Sherman and Chen, Haonan and Chandra, Abhranil and Jiang, Ziyan and Arulraj, Aaran and Wang, Kai and Do, Quy Duc and Ni, Yuansheng and Lyu, Bohan and Narsupalli, Yaswanth and Fan, Rongqi and Lyu, Zhiheng and Lin, Yuchen and Chen, Wenhu},
journal = {ArXiv},
year = {2024},
volume={abs/2406.15252},
url = {https://arxiv.org/abs/2406.15252},
}