randallcr7/OpenS2V-Eval

---
language:
- en
license: cc-by-4.0
size_categories:
- 1M<n<10M
task_categories:
- text-to-video
tags:
- subject-to-video
- text-to-video
- image-to-video
- video-generation
- large-scale
- benchmark
- evaluation
configs:
- config_name: default
data_files:
- split: open_domain
path: Open-Domain_Eval.json
- split: human_domain
path: Human-Domain_Eval.json
- split: single_domain
path: Single-Domain_Eval.json
- split: dev_hard_case
path: Hard-Case_Dev_Eval/Hard-Case_Dev_Eval.json
---

<div align=center>
<img src="https://github.com/PKU-YuanGroup/OpenS2V-Nexus/blob/main/__assets__/OpenS2V-Nexus_logo.png?raw=true" width="300px">
</div>
<h2 align="center"> <a href="https://pku-yuangroup.github.io/OpenS2V-Nexus/">OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation</a></h2>

<h5 align="center"> If you like our project, please give us a star ⭐ on GitHub for the latest update. </h5>

## ✨ Summary
**OpenS2V-Eval** introduces 180 prompts from seven major categories of S2V, which incorporate both real and synthetic test data. Furthermore,
to accurately align human preferences with S2V benchmarks, we propose three automatic metrics: **NexusScore**, **NaturalScore**, **GmeScore**
to separately quantify subject consistency, naturalness, and text relevance in generated videos. Building on this, we conduct a comprehensive
evaluation of 18 representative S2V models, highlighting their strengths and weaknesses across different content.

This benchmark is presented in the paper: [OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation](https://huggingface.co/papers/2505.20292)

## Evaluate Your Own Models
For instructions on evaluating your customized model using OpenS2V-Eval, please refer to [this guide](https://github.com/PKU-YuanGroup/OpenS2V-Nexus/tree/main/eval).

## Get Videos Generated by Different S2V models
For details on the videos generated by various S2V models, please refer to [this link](https://huggingface.co/datasets/BestWishYsh/OpenS2V-Eval/tree/main/Results).

## Description
- **Repository:** [Code](https://github.com/PKU-YuanGroup/OpenS2V-Nexus), [Page](https://pku-yuangroup.github.io/OpenS2V-Nexus/), [Dataset](https://huggingface.co/datasets/BestWishYsh/OpenS2V-5M), [Benchmark](https://huggingface.co/datasets/BestWishYsh/OpenS2V-Eval)
- **Paper:** [https://huggingface.co/papers/2505.20292](https://huggingface.co/papers/2505.20292)
- **Point of Contact:** [Shenghai Yuan]([e-mail])

## Citation
If you find our paper and code useful in your research, please consider giving a star and citation.

```BibTeX
@article{yuan2025opens2v,
title={OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation},
author={Yuan, Shenghai and He, Xianyi and Deng, Yufan and Ye, Yang and Huang, Jinfa and Lin, Bin and Luo, Jiebo and Yuan, Li},
journal={arXiv preprint arXiv:2505.20292},
year={2025}
}
```