image-to-video-human-preference-seedance-1-pro

<style> .vertical-container { display: flex; flex-direction: column; gap: 60px; } .image-container img { height: 150px; /* Set the desired height */ margin:0; object-fit: contain; /* Ensures the aspect ratio is maintained */ width: auto; /* Adjust width automatically based on height */ } .image-container { display: flex; /* Aligns images side by side */ justify-content: space-around; /* Space them evenly */ align-items: center; /* Align them vertically */ } .container { width: 90%; margin: 0 auto; } .text-center { text-align: center; display: flex; align-items:center; flex-direction: column; } .score-amount { margin: 0; margin-top: 10px; } .score-percentage { font-size: 12px; font-weight: semi-bold; } </style> # Rapidata Video Generation Hailuo-02 v Marey Human Preference <a href="https://www.rapidata.ai"> <img src="https://cdn-uploads.huggingface.co/production/uploads/66f5624c42b853e73e0738eb/jfxR79bOztqaC6_yNNnGU.jpeg" width="300" alt="Dataset visualization"> </a> <a href="https://huggingface.co/datasets/Rapidata/text-2-image-Rich-Human-Feedback"> </a> In this dataset, ~6k human responses from ~2k human annotators were collected to evaluate Seedance 1 Pro video generation model on our benchmark. This dataset was collected in roughtly 5 min using the [Rapidata Python API](https://docs.rapidata.ai), accessible to anyone and ideal for large scale data annotation. Explore our latest model rankings on our [website](https://www.rapidata.ai/benchmark). If you get value from this dataset and would like to see more in the future, please consider liking it ❤️ To add your own model to the benchmark send us an e-mail at: [e-mail] # Overview In this dataset, ~6k human responses from ~2k human annotators were collected to evaluate Seedance 1 Pro video generation model on our benchmark. This dataset was collected in roughtly 5 min using the [Rapidata Python API](https://docs.rapidata.ai), accessible to anyone and ideal for large scale data annotation. # Explanation of the colums The dataset contains paired video comparisons. Each entry includes 'video1' and 'video2' fields, which contain links to downscaled GIFs for easy viewing. The full-resolution videos can be found [here](https://huggingface.co/datasets/Rapidata/image-to-video-human-preference-seedance-1-pro/tree/main/Videos) The weighted_results column contains scores ranging from 0 to 1, representing aggregated user responses. Individual user responses can be found in the detailedResults column. # Alignment The alignment score quantifies how well an video matches its prompt. Users were asked: "Which video fits the description better?". ## Examples <div class="vertical-container"> <div class="container"> <div class="text-center"> <q>The calf moves forward, gently munching on the hay scattered on the ground.</q> <img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/DXJWSH9hHv7HXxJnPI035.jpeg" width=300> </div> <div class="image-container"> <div> <h3 class="score-amount">Seedance 1 Pro </h3> <div class="score-percentage">(Score: 90.61%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/seedance-1-pro_subject-motion_0039.gif" width=500> </div> <div> <h3 class="score-amount">Hailuo-02 </h3> <div class="score-percentage">(Score: 9.39%)</div> <img src="https://assets.rapidata.ai/hailuo-02_subject-motion_0039.gif" width=500> </div> </div> </div> <div class="container"> <div class="text-center"> <q>The fern slowly unfurls as a gentle breeze causes it to sway softly.</q> <img src="https://cdn-uploads.huggingface.co/production/uploads/672b7d79fd1e92e3c3567435/n1nLh_PI_TxqbvsH9C6FT.jpeg" width=300> </div> <div class="image-container"> <div> <h3 class="score-amount">Seedance 1 Pro </h3> <div class="score-percentage">(Score: 15.68%)</div> <img src="https://assets.rapidata.ai/seedance-1-pro_scene-motion_0060.gif" width=500> </div> <div> <h3 class="score-amount">Hailuo-02 </h3> <div class="score-percentage">(Score: 84.32%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/hailuo-02_scene-motion_0060.gif" width=500> </div> </div> </div> </div> # Coherence The coherence score measures whether the generated video is logically consistent and free from artifacts or visual glitches. Without seeing the original prompt, users were asked: "Which video has more glitches and is more likely to be AI generated?" ## Examples <div class="vertical-container"> <div class="container"> <div class="image-container"> <div> <h3 class="score-amount">Seedance 1 Pro </h3> <div class="score-percentage">(Glitch Rating: 8.78%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/seedance-1-pro_style_0047.gif" width="500" alt="Dataset visualization"> </div> <div> <h3 class="score-amount">Marey </h3> <div class="score-percentage">(Glitch Rating: 91.22%)</div> <img src="https://assets.rapidata.ai/marey_0047.gif" width="500" alt="Dataset visualization"> </div> </div> </div> <div class="container"> <div class="image-container"> <div> <h3 class="score-amount">Seedance 1 Pro </h3> <div class="score-percentage">(Glitch Rating: 80%)</div> <img src="https://assets.rapidata.ai/seedance-1-pro_style_0007.gif" width="500" alt="Dataset visualization"> </div> <div> <h3 class="score-amount">Hailuo-02 </h3> <div class="score-percentage">(Glitch Rating: 20%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/hailuo-02_style_0007.gif" width="500" alt="Dataset visualization"> </div> </div> </div> </div> # Preference The preference score reflects how visually appealing participants found each video, independent of the prompt. Users were asked: "Which video do you prefer aesthetically?" ## Examples <div class="vertical-container"> <div class="container"> <div class="image-container"> <div> <h3 class="score-amount">Seedance 1 Pro </h3> <div class="score-percentage">(Score: 90.74%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/seedance-1-pro_style_0083.gif" width="500" alt="Dataset visualization"> </div> <div> <h3 class="score-amount">Marey </h3> <div class="score-percentage">(Score: 9.26%)</div> <img src="https://assets.rapidata.ai/marey_0083.gif" width="500" alt="Dataset visualization"> </div> </div> </div> <div class="container"> <div class="image-container"> <div> <h3 class="score-amount">Seedance 1 Pro </h3> <div class="score-percentage">(Score: 9.03%)</div> <img src="https://assets.rapidata.ai/seedance-1-pro_subject-motion_0036.gif" width="500" alt="Dataset visualization"> </div> <div> <h3 class="score-amount">Hailuo-02 </h3> <div class="score-percentage">(Score: 90.97%)</div> <img style="border: 5px solid #18c54f;" src="https://assets.rapidata.ai/hailuo-02_subject-motion_0036.gif " width="500" alt="Dataset visualization"> </div> </div> </div> </div> </br> # About Rapidata Rapidata's technology makes collecting human feedback at scale faster and more accessible than ever before. Visit [rapidata.ai](https://www.rapidata.ai/) to learn more about how we're revolutionizing human feedback collection for AI development. # Other Datasets We run a benchmark of the major video generation models, the results can be found on our [website](https://www.rapidata.ai/leaderboard/video-models). We rank the models according to their coherence/plausiblity, their aligment with the given prompt and style prefernce. The underlying 2M+ annotations can be found here: - Link to the [Rich Video Annotation dataset](https://huggingface.co/datasets/Rapidata/text-2-video-Rich-Human-Feedback) - Link to the [Coherence dataset](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Coherence_Dataset) - Link to the [Text-2-Image Alignment dataset](https://huggingface.co/datasets/Rapidata/Flux_SD3_MJ_Dalle_Human_Alignment_Dataset) - Link to the [Preference dataset](https://huggingface.co/datasets/Rapidata/700k_Human_Preference_Dataset_FLUX_SD3_MJ_DALLE3)