mt_bench_prompts

# MT Bench by LMSYS
This set of evaluation prompts is created by the [LMSYS org](https://huggingface.co/lmsys) for better evaluation of chat models.
For more information, see the [paper](https://arxiv.org/abs/2306.05685).

### Dataset loading
To load this dataset, use 🤗 datasets:
```python
from datasets import load_dataset
data = load_dataset(HuggingFaceH4/mt_bench_prompts, split="train")
```
### Dataset creation
To create the dataset, we do the following for our internal tooling.
* rename `turns` to `prompts`,
* add empty `reference` to remaining prompts (for HF Datasets),
* Use the following code to load and save as a dataset
```python
from datasets import load_dataset
import hashlib

data = load_dataset("json", data_files="https://huggingface.co/datasets/HuggingFaceH4/mt_bench_prompts/raw/main/raw/question.jsonl", split="train")

# %% create_dataset.ipynb 11
def format_example(example):
return {
"prompt": example["prompt"],
"prompt_id": int(hashlib.sha256(''.join(example["prompt"]).encode("utf-8")).hexdigest(), 16) % (10 ** 8),
"category": example["category"],
"reference": example["reference"],
}

formatted_ds = data.map(format_example, num_proc=6, remove_columns=data.column_names)

#
formatted_ds.push_to_hub("HuggingFaceH4/mt_bench_prompts", split="train")
```

# LMSYS 开发的MT Bench评测数据集
本评测提示集由[LMSYS组织](https://huggingface.co/lmsys)开发，旨在优化对话模型的评估流程。如需了解更多细节，请参阅相关[论文](https://arxiv.org/abs/2306.05685)。

### 数据集加载
若需加载本数据集，请使用🤗 数据集库：
python
from datasets import load_dataset
data = load_dataset("HuggingFaceH4/mt_bench_prompts", split="train")

### 数据集构建
若需基于内部工具构建该数据集，请执行以下操作：
* 将字段`turns`重命名为`prompts`；
* 为剩余提示项添加空的`reference`字段（适配HF数据集库）；
* 使用以下代码加载并封装为标准数据集：
python
from datasets import load_dataset
import hashlib

data = load_dataset("json", data_files="https://huggingface.co/datasets/HuggingFaceH4/mt_bench_prompts/raw/main/raw/question.jsonl", split="train")

# %% create_dataset.ipynb 11
def format_example(example):
return {
"prompt": example["prompt"],
"prompt_id": int(hashlib.sha256(''.join(example["prompt"]).encode("utf-8")).hexdigest(), 16) % (10 ** 8),
"category": example["category"],
"reference": example["reference"],
}

formatted_ds = data.map(format_example, num_proc=6, remove_columns=data.column_names)

#
formatted_ds.push_to_hub("HuggingFaceH4/mt_bench_prompts", split="train")