Replication Data for: What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases

The OLIVE dataset is a highly diverse, human-corrected multi-modal collection designed to simulate the variety and idiosyncrasies of user queries vision-language models (VLMs) face in real-world scenarios. It supports the training and evaluation of VLMs in conditions that more closely resemble their ultimate use cases. The dataset contains 9,450 images, 30,120 unique instructions, and 47,250 responses. Each image corresponds with five instruction-response pairs. The instruction-response pairs can be broadly categorised into four groups: visual recognition, creative writing, knowledge-based, and elaborated description.