ColorBench: Replication Package for "Trust-Calibration Failures of Multimodal LLMs in Identity-Sensitive Beauty Color Recommendation"

Replication package for the paper "Trust-Calibration Failures of Multimodal LLMs in Identity-Sensitive Beauty Color Recommendation: A Cross-Model Analysis" (Li & Ren). ColorBench is a trust-centered diagnostic evaluation of four multimodal large language models (Qwen2-VL-7B, Qwen-VL-Max, GLM-4.6V, GPT-4o) on identity-sensitive seasonal beauty color recommendation, a subjective task with no validated ground truth. This package contains everything needed to reproduce the study from the original FairFace dataset. Contents:- data/sampling_indices.csv — the 560 FairFace training-set indices used (7 race groups x 2 genders x 40), demographically balanced- data/model_outputs/ — raw per-image outputs of all four models under the frozen prompt (560 images each)- data/ summaries — aggregate and per-model behavior statistics- data/human_study/ — de-identified expert-survey data (N=346) and rating-study data (3 raters x 50 images x 3 models), with a codebook- prompts/frozen_prompt.txt — the single neutral prompt applied verbatim and identically to all four models- code/ — sampling, inference, and analysis scripts (API keys are read from environment variables; none are included) Face images are NOT redistributed. The 560 images are sourced from the FairFace dataset (Kärkkäinen & Joo, WACV 2021, CC BY 4.0); we provide the exact sampling indices so the sample can be reconstructed from the original dataset under its own license. Survey and rating data are released in de-identified form (no names, IP addresses, or timestamps). License: code under MIT; data under CC BY 4.0. This work is currently under peer review; citation metadata will be updated upon publication.