openmodelinitiative/Rapidata-Finding_the_Subjective_Truth

---
license: other
license_name: cdla-permissive-2.0
license_link: https://cdla.dev/permissive-2-0/
---

This data was provided by Rapidata for open sourcing by the Open Model Initiative
You can learn more about Rapidata's global preferencing & human labeling solutions at https://rapidata.ai/

This folder contains the data behind the paper "Finding the subjective Truth - Collecting 2 million votes for comprehensive gen-ai model evaluation"
The paper can be found here: https://arxiv.org/html/2409.11904

Rapidata-Benchmark_v1.0.tsv: Contains the 282 prompts that were used to generate the images with each of the evaluated models.

Generated_Images: Contains all the generated images that were evaluated. For each prompt from "Rapidata-Benchmark_v1.0.tsv", 4 images were generated with every evaluated model.

The 2M+ human annotations can be found split into 3 different queries in these folders:
-Rapidata_Preference:
-dalle3_vs_mj_preference.json
-dalle3_vs_std_preference.json
....
-Rapidata_Coherence:
-dalle3_vs_mj_coherence.json
-dalle3_vs_std_coherence.json
....
-Rapidata_Text_Image_Alignment:
-dalle3_vs_mj_text_alignment.json
-dalle3_vs_std_text_alignment.json
....
All results that do NOT contain flux contain additional meta data. If there is popular demand we can add this meta data to the flux results as well.

Flux1.0_vs_Flux1.1Blueberry: Contains independent results for a comparison between the two Black Forest Lab models with the same setup.