KlingTeam/VideoGen-RewardBench

---
license: apache-2.0
configs:
- config_name: default
data_files:
- split: eval
path: "videogen-rewardbench.csv"
---

<div align="center">
<p align="center">
🏆 <a href="https://huggingface.co/spaces/KwaiVGI/VideoGen-RewardBench" target="_blank">[VideoGen-RewardBench Leaderboard]</a>
</p>
</div>

## Introduction
**VideoGen-RewardBench** is a comprehensive benchmark designed to evaluate the performance of video reward models on modern text-to-video (T2V) systems. Derived from the third-party [VideoGen-Eval](https://github.com/AILab-CVC/VideoGen-Eval/tree/main) (Zeng et.al, 2024), we constructing 26.5k (prompt, Video A, Video B) triplets and employing expert annotators to provide pairwise preference labels.

These annotations are based on key evaluation dimensions—**Visual Quality (VQ)**, **Motion Quality (MQ)**, **Temporal Alignment (TA)**, and an overall quality score—ensuring a nuanced assessment of each generated video. It covers a diverse range of prompts and videos generated by 12 state-of-the-art T2V models, featuring high resolutions (480×720 to 576×1024) as well as longer durations (4s to 6s). VideoGen-RewardBench offers a robust and fair evaluation framework that accurately reflects human preferences and the latest advancements in video generation.

## Dataset Structure
### Data Instances

An example looks as follows:
```json
{
"path_A": "videos/kling1.5/kling1.5_00103.mp4",
"path_B": "videos/minimax/minimax_00103.mp4",
"A_model": "kling1.5",
"B_model": "minimax",
"prompt": "Static camera, a metal ball rolls on a smooth tabletop.",
"VQ": "A",
"MQ": "A",
"TA": "A",
"Overall": "A",
"fps_A" :30.0,
"num_frames_A": 153.0,
"fps_B": 25.0,
"num_frames_B": 141.0,
}
```

### Data Fields
The data fields are:

- `path_A`: The file path of Video A in the pair.
- `path_B`: The file path of Video B in the pair.
- `A_model`: The name of the model that generated Video A.
- `B_model`: The name of the model that generated Video B.
- `prompt`: The text prompt used to generate both videos.
- `VQ`: The video with better visual quality between video A and video B.
- `MQ`: The video with better motion quality between video A and video B.
- `TA`: The video with better text alignment between video A and video B.
- `Overall`: The video with better overall quality between video A and video B.
- `fps_A`: The FPS of Video A.
- `num_frames_A`: The number of frames in Video A.
- `fps_B`: The FPS of Video B.
- `num_frames_B`: The number of frames in Video B.

## Citation
If you find this project useful, please consider citing:
```bibtex
@article{liu2025improving,
title={Improving Video Generation with Human Feedback},
author={Jie Liu and Gongye Liu and Jiajun Liang and Ziyang Yuan and Xiaokun Liu and Mingwu Zheng and Xiele Wu and Qiulin Wang and Wenyu Qin and Menghan Xia and Xintao Wang and Xiaohong Liu and Fei Yang and Pengfei Wan and Di Zhang and Kun Gai and Yujiu Yang and Wanli Ouyang},
journal={arXiv preprint arXiv:2501.13918},
year={2025}
}