ledengary/VLCB

VLCB（视觉-语言置信度基准）是一个用于评估大型视觉-语言模型置信度估计方法的基准测试数据集。它汇集了来自七个公开可用的视觉问答数据集的30,514个测试问题，并为每个问题提供了五个开源权重的大型视觉-语言模型的逐字贪婪输出，以及由统一的LLM法官（gpt-5-mini）分配的二元正确性标签。数据集仅包含模型输出和正确性标签，不包含原始图像和问题；用户需要从原始分发者处获取源数据，并通过确定性哈希ID在本地重建完整数据集。数据集分为训练集、验证集和测试集，分别包含20,000、5,000和30,514个唯一样本，行数分别为100,000、25,000和152,568。测试集由GQA、POPE、GMAI-MMBench、MMMU-Pro（10选项和4选项）、MME-Finance和LLaVA-in-the-Wild组成。使用的模型包括Qwen3-VL-8B-Instruct、llava-v1.6-vicuna-13b-hf、InternVL3_5-14B-HF、gemma-3-27b-it和deepseek-vl2。数据集旨在促进对LVLM置信度估计、校准、不确定性和幻觉问题的研究，适用于非商业研究用途，受源数据集许可限制。VLCB (Vision-Language Confidence Benchmark) is a benchmark for evaluating confidence estimation methods on Large Vision-Language Models (LVLMs). It pools 30,514 test questions from seven publicly available VQA datasets and, for each question, ships the verbatim greedy outputs of five open-weight LVLMs together with binary correctness labels assigned by a unified LLM judge (gpt-5-mini). The dataset releases only the model outputs and correctness labels; users must obtain the original images and questions from the source distributors and reconstruct VLCB locally using a deterministic hash_id. The dataset is split into train, validation, and test sets, with 20,000, 5,000, and 30,514 unique samples, and row counts of 100,000, 25,000, and 152,568, respectively. The test set comprises GQA, POPE, GMAI-MMBench, MMMU-Pro (10-option and 4-option), MME-Finance, and LLaVA-in-the-Wild. The models used include Qwen3-VL-8B-Instruct, llava-v1.6-vicuna-13b-hf, InternVL3_5-14B-HF, gemma-3-27b-it, and deepseek-vl2. It is designed for research on LVLM confidence estimation, calibration, uncertainty, and hallucination, and is intended for non-commercial research use only, subject to source dataset licenses.