text-2-video-ranking-human-preferences

# T2V Ranking Human Preferences <p align="left"> <img src="https://huggingface.co/datasets/datapointai/text-2-video-ranking-human-preferences/resolve/main/datapointlogo.png" alt="Datapoint AI" width="300"> </p> **~91,000 human ranking labels** across **18 text-to-video models** on **3 quality dimensions**, collected from real annotators via [Datapoint AI](https://trydatapoint.com). This is the first public **ranking-based** (not pairwise) human preference dataset for text-to-video generation. Each datapoint contains **5 videos** generated from the same prompt by different models, ranked 1st through 5th by **15 annotators** on each dimension. ## Why This Dataset Existing video preference datasets use pairwise comparisons (A vs B). Ranking 5 models simultaneously is richer: you get a full ordering per annotator, enabling: - **Ranking-based reward models** (not just binary preferences) - **Listwise preference optimization** (beyond DPO's pairwise paradigm) - **Per-dimension analysis** — visual quality, prompt adherence, and physical realism are scored separately - **Cross-model leaderboards** with statistical significance from 15 annotators per datapoint ## Models Compared 18 text-to-video models from 2024-2025. Quality tiers are as defined in [VideoFeedback2 (He et al., 2025)](https://arxiv.org/abs/2509.22799), Table 8: | Tier | Models | |------|--------| | **Perfect/Modern** | Kling-1.6, Sora, StepVideo-T2V, Pika-2.2, Wanx-2.1 (14B), Ruyi, CogVideoX-1.5 | | **Good** | CogVideoX 5B, Mochi1-Preview, MagicTime, Wanx-2.1 (1.3B), OpenSora-Plan v1.3 | | **Moderate** | LTX-Video 0.9.5, CogVideoX 2B, Latte, OpenSora v1.2, VideoCrafter2, Vchitect-2.0 | Each datapoint samples 5 models (1 Perfect/Modern + 2 Good + 2 Moderate) to ensure tier diversity. ## Key Findings ### Overall Leaderboard (weighted across all 3 dimensions) | Rank | Model | Avg Rank | Win Rate | Tier | |------|-------|----------|----------|------| | 1 | **Kling-1.6** | 2.384 | 60.7% | Perfect/Modern | | 2 | **Sora** | 2.424 | 58.2% | Perfect/Modern | | 3 | **StepVideo-T2V** | 2.435 | 59.2% | Perfect/Modern | | 4 | **Pika-2.2** | 2.613 | 40.4% | Perfect/Modern | | 5 | **Wanx-2.1 (14B)** | 2.741 | 31.6% | Perfect/Modern | | 6 | Mochi1-Preview | 2.742 | 30.1% | Good | | 7 | CogVideoX 5B | 2.795 | 25.3% | Good | | 8 | CogVideoX-1.5 | 2.801 | 30.3% | Perfect/Modern | | 9 | Wanx-2.1 (1.3B) | 2.906 | 20.2% | Good | | 10 | Ruyi | 3.004 | 29.9% | Perfect/Modern | | 11 | OpenSora-Plan v1.3 | 3.126 | 9.7% | Good | | 12 | LTX-Video 0.9.5 | 3.138 | 10.9% | Moderate | | 13 | CogVideoX 2B | 3.195 | 8.3% | Moderate | | 14 | Latte | 3.215 | 7.2% | Moderate | | 15 | OpenSora v1.2 | 3.256 | 7.6% | Moderate | | 16 | VideoCrafter2 | 3.337 | 5.2% | Moderate | | 17 | Vchitect-2.0 | 3.356 | 4.5% | Moderate | | 18 | MagicTime | 3.374 | 4.8% | Good | ### Dimension Highlights - **Kling-1.6** dominates visual quality with a 69.6% win rate - **Sora** leads text alignment with 58.6% win rate - **Ruyi** is an outlier: classified as Perfect/Modern but places 10th overall due to poor text alignment (often ignores the prompt entirely) - **MagicTime** underperforms its Good tier, ranking below all Moderate models ## Dataset Structure 2009 rows, each with 5 ranked videos and aggregated annotations. ### Columns | Column | Type | Description | |--------|------|-------------| | `prompt` | string | Text prompt used for video generation | | `video_1` .. `video_5` | Video | MP4 videos from 5 different models | | `model_1` .. `model_5` | string | Model name for each video | | `tier_1` .. `tier_5` | string | Quality tier per [VideoFeedback2](https://arxiv.org/abs/2509.22799) (Perfect/Modern, Good, Moderate) | | `visual_quality_avg_ranks` | float[5] | Mean rank per video (1=best, 5=worst) from 15 annotators | | `visual_quality_ranking_order` | int[5] | Videos sorted best to worst | | `visual_quality_total_responses` | int | Number of annotator rankings | | `text_alignment_avg_ranks` | float[5] | Same for text-to-video alignment | | `text_alignment_ranking_order` | int[5] | | | `text_alignment_total_responses` | int | | | `physical_consistency_avg_ranks` | float[5] | Same for physical/common-sense realism | | `physical_consistency_ranking_order` | int[5] | | | `physical_consistency_total_responses` | int | | | `overall_weighted_ranks` | float[5] | Weighted average rank (0.334 visual + 0.333 text + 0.333 physical) | | `overall_best` | int | Index (1-5) of the best video overall | ### Rank Convention - `avg_ranks[i]` corresponds to `video_{i+1}` / `model_{i+1}` - Lower rank = better (1.0 is best possible, 5.0 is worst) - `ranking_order` lists video indices sorted best to worst ## Usage ```python from datasets import load_dataset ds = load_dataset("datapointai/text-2-video-ranking-human-preferences", split="train") row = ds[0] print(row["prompt"]) best = row["overall_best"] print(f"Best overall: video_{best} ({row[f'model_{best}']})") # Per-dimension rankings for dim in ["visual_quality", "text_alignment", "physical_consistency"]: order = row[f"{dim}_ranking_order"] print(f" {dim}: {[row[f'model_{i}'] for i in order]}") ``` ### Filter by Model ```python # Find all datapoints where Sora appears sora_rows = ds.filter( lambda x: any(x[f"model_{i}"] == "Sora" for i in range(1, 6)) ) print(f"{len(sora_rows)} datapoints with Sora") ``` ### Train a Ranking Reward Model ```python for row in ds: prompt = row["prompt"] # 5 videos with their overall weighted ranks (lower = better) items = [ (row[f"video_{i}"], row[f"model_{i}"], row["overall_weighted_ranks"][i-1]) for i in range(1, 6) ] items.sort(key=lambda x: x[2]) # sort by rank # items[0] is the best, items[-1] is the worst # Use for listwise ranking loss, or extract pairwise preferences ``` ## Data Collection ### Source Videos Videos are sourced from the [VideoFeedback2](https://huggingface.co/datasets/TIGER-Lab/VideoFeedback2) dataset (Apache 2.0, [He et al., 2025](https://arxiv.org/abs/2509.22799)), which contains 27K AI-generated videos from 22 text-to-video models across 2,933 prompts. We filtered to 18 models released in 2024 or later. ### Sampling Strategy For each prompt, we sampled 5 models using stratified selection: **1 Perfect/Modern + 2 Good + 2 Moderate** tier models (tiers per [VideoFeedback2, Table 8](https://arxiv.org/abs/2509.22799)). This ensures every datapoint spans a meaningful quality range while keeping the ranking task manageable for annotators. ### Annotation Rankings were collected through [Datapoint AI](https://trydatapoint.com)'s annotation platform: - **15 annotators per datapoint** per dimension - **3 dimensions**: visual quality, text-to-video alignment, physical/common-sense consistency - Videos were **shuffled** per session to prevent position bias - Annotators ranked all 5 videos from best to worst for each dimension - Total: **~91,000 individual ranking labels** ### Aggregation Per-dimension results are aggregated via **mean rank** across 15 annotators (Borda count). The overall score is a weighted average: 33.4% visual quality + 33.3% text alignment + 33.3% physical consistency. ## License CC-BY-4.0 ## Citation If you use this dataset, please cite both this dataset and the source video data: ```bibtex @dataset{datapointai_t2v_ranking_2026, title={T2V Ranking Human Preferences: 18 Models, 91K Rankings, 3 Dimensions}, author={Datapoint AI}, year={2026}, url={https://huggingface.co/datasets/datapointai/text-2-video-ranking-human-preferences}, note={~91,000 ranking-based human preference labels for text-to-video generation across 18 models and 3 quality dimensions} } @misc{he2025videoscore2, title={VideoScore2: Think before You Score in Generative Video Evaluation}, author={Xuan He and Dongfu Jiang and Ping Nie and others}, year={2025}, eprint={2509.22799}, archivePrefix={arXiv}, primaryClass={cs.CV} } ```