mair-lab/lave-human-feedback

--- dataset_info: features: - name: dataset dtype: string - name: model dtype: string - name: qid dtype: int64 - name: question dtype: string - name: references sequence: string - name: prediction dtype: string - name: human_score dtype: float64 splits: - name: dev num_bytes: 930874 num_examples: 7000 - name: test num_bytes: 3774759 num_examples: 22050 download_size: 1623960 dataset_size: 4705633 configs: - config_name: default data_files: - split: dev path: data/dev-* - split: test path: data/test-* license: cc-by-4.0 tags: - human-feedback --- # LAVE human judgments This repository contains the human judgment data for [Improving Automatic VQA Evaluation Using Large Language Models](https://arxiv.org/abs/2310.02567). Details about the data collection process and crowdworker population can be found in our paper, specifically in section 5.2 and appendix A.1. Fields: * **dataset:** VQA dataset of origin for this example (`vqav2`, `vgqa`, `okvqa`). * **model:** VQA model that generated the predicted answer (`blip2`, `promptcap`, `blip_vqa`, `blip_vg`). * **qid:** question ID coming from the original dataset. * **question**: question copied from the original dataset for convenience. * **references:** reference answers copied from the original dataset for convenience. * **prediction:** candidate answer generated by the VQA model. * **human_score:** human judgment score, with `0` meaning incorrect answer, `0.5` ambiguous or incomplete answer, and `1` correct answer. ## Usage ```python from datasets import load_dataset # Load the dev split dataset = load_dataset("mair-lab/lave-human-feedback", split="dev") # Filter examples by dataset and model dataset = dataset.filter(lambda example: example["dataset"] == "vqav2" and example["model"] == "blip2") ```

原始信息汇总

数据集概述

数据集特征

• dataset: 字符串类型，原始VQA数据集来源（vqav2, vgqa, okvqa）。
• model: 字符串类型，生成预测答案的VQA模型（blip2, promptcap, blip_vqa, blip_vg）。
• qid: 整数类型，原始数据集中的问题ID。
• question: 字符串类型，从原始数据集中复制的问题。
• references: 字符串序列类型，从原始数据集中复制的参考答案。
• prediction: 字符串类型，VQA模型生成的候选答案。
• human_score: 浮点数类型，人类判断分数，其中0表示答案错误，0.5表示答案模糊或不完整，1表示答案正确。

数据集拆分

• dev: 包含7000个示例，总大小为930874字节。
• test: 包含22050个示例，总大小为3774759字节。

数据集大小

• 下载大小: 1623960字节。
• 数据集总大小: 4705633字节。

数据文件配置

• config_name: default
• data_files:
  • split: dev, path: data/dev-*
  • split: test, path: data/test-*

许可证

• cc-by-4.0

标签

• human-feedback