Rendra86318/vlms-are-biased

VLMBias数据集包含跨越7个不同领域的图像-问题对：动物、标志、国旗、象棋棋子、棋盘游戏、光学错觉和图案网格。每个领域都提供了经过微妙修改的反事实图像，旨在测试真正的视觉计数和推理能力，而非依赖记忆的偏见。数据集包括计数对象部分（如腿、条纹、星星、棋子、网格线）和识别异常或变化等任务。该数据集揭示了最先进的视觉语言模型在计数任务中依赖记忆知识而非真实视觉分析的严重缺陷。

The VLMBias dataset comprises image-question pairs across 7 diverse domains: Animals, Logos, National Flags, Chess Pieces, Board Games, Optical Illusions, and Patterned Grids. For each domain, we provide counterfactual images with subtle modifications designed to test genuine visual counting and reasoning against memorized biases. The dataset includes tasks such as counting object parts (e.g., legs, stripes, stars, pieces, grid lines) and identifying anomalies or changes. This dataset reveals a critical flaw in state-of-the-art Vision Language Models (VLMs), showing their strong reliance on memorized knowledge rather than genuine visual analysis in counting tasks.