lmsys/mt_bench_human_judgments|自然语言处理数据集|机器学习模型评估数据集

---
dataset_info:
features:
- name: question_id
dtype: int64
- name: model_a
dtype: string
- name: model_b
dtype: string
- name: winner
dtype: string
- name: judge
dtype: string
- name: conversation_a
list:
- name: content
dtype: string
- name: role
dtype: string
- name: conversation_b
list:
- name: content
dtype: string
- name: role
dtype: string
- name: turn
dtype: int64
splits:
- name: human
num_bytes: 15003469
num_examples: 3355
- name: gpt4_pair
num_bytes: 10679650
num_examples: 2400
download_size: 1388888
dataset_size: 25683119
license: cc-by-4.0
task_categories:
- conversational
- question-answering
language:
- en
size_categories:
- 1K<n<10K
---

## Content
This dataset contains 3.3K expert-level pairwise human preferences for model responses generated by 6 models in response to 80 MT-bench questions.
The 6 models are GPT-4, GPT-3.5, Claud-v1, Vicuna-13B, Alpaca-13B, and LLaMA-13B. The annotators are mostly graduate students with expertise in the topic areas of each of the questions. The details of data collection can be found in our [paper](https://arxiv.org/abs/2306.05685).

## Agreement Calculation
This Colab [notebook](https://colab.research.google.com/drive/1ctgygDRJhVGUJTQy8-bRZCl1WNcT8De6?usp=sharing) shows how to compute the agreement between humans and GPT-4 judge with the dataset. Our results show that humans and GPT-4 judge achieve over 80\% agreement, the same level of agreement between humans.

## Citation
```
@misc{zheng2023judging,
title={Judging LLM-as-a-judge with MT-Bench and Chatbot Arena},
author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric. P Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},
year={2023},
eprint={2306.05685},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

原始信息汇总

数据集概述

数据集信息

• 特征（Features）:

  • question_id: 数据类型为 int64
  • model_a: 数据类型为 string
  • model_b: 数据类型为 string
  • winner: 数据类型为 string
  • judge: 数据类型为 string
  • conversation_a: 包含两个子特征
    • content: 数据类型为 string
    • role: 数据类型为 string
  • conversation_b: 包含两个子特征
    • content: 数据类型为 string
    • role: 数据类型为 string
  • turn: 数据类型为 int64

• 数据分割（Splits）:

  • human: 大小为 15003469 字节，包含 3355 个样本
  • gpt4_pair: 大小为 10679650 字节，包含 2400 个样本

• 下载大小（Download Size）: 1388888 字节

• 数据集大小（Dataset Size）: 25683119 字节

• 许可证（License）: cc-by-4.0

• 任务类别（Task Categories）:

  • 对话（conversational）
  • 问答（question-answering）

• 语言（Language）: 英语（en）

• 大小类别（Size Categories）: 1K<n<10K

数据集内容

• 包含 3.3K 专家级的人类偏好数据，用于评估 6 个模型对 80 个 MT-bench 问题的响应。
• 涉及的模型包括 GPT-4, GPT-3.5, Claud-v1, Vicuna-13B, Alpaca-13B, 和 LLaMA-13B。
• 注释者主要是具有相关问题领域专业知识的研究生。

数据集用途

• 用于计算人类与 GPT-4 判断者之间的协议，显示两者之间的协议度超过 80%。