vist

# VIST: Visual Style Transfer Benchmark VIST is a human-based benchmark for evaluating style transfer in image generation. ## Dataset Description This benchmark includes: - Reference images with original style - Prompts for image generation - Generated images produced by 12 systems - Human annotation results with pairwise A/B evaluations Systems under evaluation are expected to generate images by prompt conditioned on the reference images. Their outputs are then compared against each other. ## Dataset Structure ``` toloka/vist/ ├── data/test.parquet # Annotations with image paths (39,300 rows) ├── output_images/ # Generated images (720 files) │ ├── Exactly/ │ ├── Firefly/ │ └── ... └── reference_images/ # Reference style images (5 unique) ├── Revolut.png ├── Dropbox.png └── ... ``` ## Usage ```python from datasets import load_dataset from huggingface_hub import hf_hub_download from PIL import Image # Load dataset ds = load_dataset("toloka/vist") # Get a sample sample = ds["test"][0] print(sample["prompt"]) print(sample["winner"]) # Load images ref_img = Image.open(hf_hub_download("toloka/vist", sample["ref_image"], repo_type="dataset")) system_a_img = Image.open(hf_hub_download("toloka/vist", sample["system_a_image"], repo_type="dataset")) system_b_img = Image.open(hf_hub_download("toloka/vist", sample["system_b_image"], repo_type="dataset")) ``` ## Evaluation Criteria The benchmark assesses style transfer quality across five key criteria: 1. **Colour palette and tonal balance** 2. **Composition and layout** (framing, perspective, placement of elements) 3. **Brand elements** (e.g., logo accuracy, iconography, type/lettering if present) 4. **Textures and line quality** (brush/pen strokes, grain, edge fidelity) 5. **Human depiction** (if present; how facial features, hands, and body parts are drawn) ## Dataset Fields | Field | Type | Description | |-------|------|-------------| | `batch_id` | int | Batch identifier | | `query_id` | int | Query identifier | | `annotator_id` | int | Annotator's identifier | | `annotator_occupation` | string | Annotator's occupation | | `prompt` | string | Text prompt for image generation | | `style` | string | Style base (brand name) | | `ref_image` | string | Path to reference style image | | `system_a` | string | Name of system A | | `system_b` | string | Name of system B | | `system_a_image` | string | Path to image generated by system A | | `system_b_image` | string | Path to image generated by system B | | `winner` | string | Winner system name | ## Systems Evaluated The benchmark includes outputs from 12 image generation systems: - Exactly.ai - Firefly Image 4 Ultra - Freepik - GPT (DALL-E) - Kontext - Krea - Leonardo.ai - Nano-Banana (Gemini 2.5 Flash Image) - OpenArt - Qwen-Image-Edit - Recraft - Wixel