HuggingFaceH4/mt_bench_prompts

---
license: apache-2.0
task_categories:
- question-answering
- conversational
language:
- en
tags:
- evaluation
pretty_name: MT Bench
size_categories:
- n<1K
---

# MT Bench by LMSYS
This set of evaluation prompts is created by the [LMSYS org](https://huggingface.co/lmsys) for better evaluation of chat models.
For more information, see the [paper](https://arxiv.org/abs/2306.05685).

### Dataset loading
To load this dataset, use 🤗 datasets:
```python
from datasets import load_dataset
data = load_dataset(HuggingFaceH4/mt_bench_prompts, split="train")
```
### Dataset creation
To create the dataset, we do the following for our internal tooling.
* rename `turns` to `prompts`,
* add empty `reference` to remaining prompts (for HF Datasets),
* Use the following code to load and save as a dataset
```python
from datasets import load_dataset
import hashlib

data = load_dataset("json", data_files="https://huggingface.co/datasets/HuggingFaceH4/mt_bench_prompts/raw/main/raw/question.jsonl", split="train")

# %% create_dataset.ipynb 11
def format_example(example):
return {
"prompt": example["prompt"],
"prompt_id": int(hashlib.sha256(''.join(example["prompt"]).encode("utf-8")).hexdigest(), 16) % (10 ** 8),
"category": example["category"],
"reference": example["reference"],
}

formatted_ds = data.map(format_example, num_proc=6, remove_columns=data.column_names)

#
formatted_ds.push_to_hub("HuggingFaceH4/mt_bench_prompts", split="train")
```

原始信息汇总

数据集概述

基本信息

• 许可证: Apache-2.0
• 任务类别:
  • 问答
  • 对话
• 语言: 英语
• 标签: 评估
• 美观名称: MT Bench
• 大小类别: 小于1K

数据集创建与加载

• 创建过程:
  • 重命名 turns 为 prompts
  • 为剩余的提示添加空的 reference
  • 使用特定代码加载并保存为数据集
• 加载方式:
  • 使用 🤗 datasets 库加载数据集 python from datasets import load_dataset data = load_dataset("HuggingFaceH4/mt_bench_prompts", split="train")

搜集汇总

背景与挑战

背景概述

该数据集是一个由LMSYS组织创建的多轮对话评估数据集，包含80个英文提示，覆盖写作、推理、数学、编程等8个类别，旨在通过多样化任务全面评估聊天模型的性能。数据集基于论文arxiv:2306.05685构建，采用Apache-2.0许可证，以parquet格式提供，适用于问答任务。