toloka/vist

VIST是一个基于人类的基准测试，用于评估图像生成中的风格转换。该基准测试包括：带有原始风格的参考图像、用于图像生成的提示、由12个系统生成的图像以及带有成对A/B评估的人类标注结果。评估的系统需要根据参考图像生成图像，然后将其输出相互比较。数据集结构包括测试数据、生成的图像和参考图像。评估标准涵盖五个关键方面：调色板和色调平衡、构图和布局、品牌元素、纹理和线条质量以及人类描绘（如果存在）。数据集字段包括批次ID、查询ID、标注者ID、标注者职业、提示、风格基础、参考图像路径、系统A和B的名称、系统A和B生成的图像路径以及获胜系统名称。参与评估的系统包括Exactly.ai、Firefly Image 4 Ultra、Freepik等12个系统。

VIST is a human-based benchmark for evaluating style transfer in image generation. This benchmark includes: reference images with original style, prompts for image generation, generated images produced by 12 systems, and human annotation results with pairwise A/B evaluations. Systems under evaluation are expected to generate images by prompt conditioned on the reference images. Their outputs are then compared against each other. The dataset structure includes test data, generated images, and reference images. Evaluation criteria assess style transfer quality across five key aspects: colour palette and tonal balance, composition and layout, brand elements, textures and line quality, and human depiction (if present). Dataset fields include batch ID, query ID, annotator ID, annotator occupation, prompt, style base, reference image path, names of systems A and B, paths to images generated by systems A and B, and winner system name. Systems evaluated include Exactly.ai, Firefly Image 4 Ultra, Freepik, and 9 others.