yzygalaxy/mt_bench_human_judgments

--- dataset_info: features: - name: question_id dtype: int64 - name: model_a dtype: string - name: model_b dtype: string - name: winner dtype: string - name: judge dtype: string - name: conversation_a list: - name: content dtype: string - name: role dtype: string - name: conversation_b list: - name: content dtype: string - name: role dtype: string - name: turn dtype: int64 splits: - name: human num_bytes: 15003469 num_examples: 3355 - name: gpt4_pair num_bytes: 10679650 num_examples: 2400 download_size: 1388888 dataset_size: 25683119 license: cc-by-4.0 task_categories: - conversational - question-answering language: - en size_categories: - 1K<n<10K --- ## Content This dataset contains 3.3K expert-level pairwise human preferences for model responses generated by 6 models in response to 80 MT-bench questions. The 6 models are GPT-4, GPT-3.5, Claud-v1, Vicuna-13B, Alpaca-13B, and LLaMA-13B. The annotators are mostly graduate students with expertise in the topic areas of each of the questions. The details of data collection can be found in our [paper](https://arxiv.org/abs/2306.05685). ## Agreement Calculation This Colab [notebook](https://colab.research.google.com/drive/1ctgygDRJhVGUJTQy8-bRZCl1WNcT8De6?usp=sharing) shows how to compute the agreement between humans and GPT-4 judge with the dataset. Our results show that humans and GPT-4 judge achieve over 80\% agreement, the same level of agreement between humans. ## Citation ``` @misc{zheng2023judging, title={Judging LLM-as-a-judge with MT-Bench and Chatbot Arena}, author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric. P Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica}, year={2023}, eprint={2306.05685}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```数据集信息 ## 特征 - 名称：问题ID（question_id），数据类型：int64 - 名称：模型A（model_a），数据类型：字符串 - 名称：模型B（model_b），数据类型：字符串 - 名称：获胜模型（winner），数据类型：字符串 - 名称：评判者（judge），数据类型：字符串 - 名称：对话A（conversation_a），列表类型，每个元素包含： - 内容（content）：字符串类型 - 角色（role）：字符串类型 - 名称：对话B（conversation_b），列表类型，每个元素包含： - 内容（content）：字符串类型 - 角色（role）：字符串类型 - 名称：轮次（turn），数据类型：int64 ## 数据划分 - 划分名称：human，字节数：15003469，样本数：3355 - 划分名称：gpt4_pair，字节数：10679650，样本数：2400 下载大小：1388888 字节，数据集总大小：25683119 字节 许可协议：知识共享署名4.0（CC-BY-4.0） 任务类别：对话式、问答 语言：英语 规模类别：1K < n < 10K --- ## 内容 本数据集涵盖3300条专家级成对人类偏好标注，对应6个模型针对80个MT-bench问题生成的模型回复。参与对比的6个模型分别为GPT-4、GPT-3.5、Claud-v1、Vicuna-13B、Alpaca-13B及LLaMA-13B。标注人员大多为具备对应问题主题领域专业知识的研究生。数据收集的详细细节可参阅我们的[论文](https://arxiv.org/abs/2306.05685)。 ## 一致性计算 本Colab[笔记](https://colab.research.google.com/drive/1ctgygDRJhVGUJTQy8-bRZCl1WNcT8De6?usp=sharing)展示了如何基于本数据集计算人类标注者与GPT-4评判器之间的一致性。实验结果表明，人类与GPT-4评判器的一致性超过80%，与人类标注者之间的一致性水平相当。 ## 引用 @misc{zheng2023judging, title={Judging 大语言模型（Large Language Model）-as-a-judge with MT-Bench and Chatbot Arena}, author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric. P Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica}, year={2023}, eprint={2306.05685}, archivePrefix={arXiv}, primaryClass={cs.CL} }