PPE-MMLU-Pro-Best-of-K

# Overview

This contains the MMLU-Pro correctness preference evaluation set for Preference Proxy Evaluations.

The prompts are sampled from [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro).

This dataset is meant for benchmarking and evaluation, not for training.

[Paper](https://arxiv.org/abs/2410.14872)

[Code](https://github.com/lmarena/PPE)

# License

User prompts are licensed under MIT, and model outputs are governed by the terms of use set by the respective model providers.

# Citation

```
@misc{frick2024evaluaterewardmodelsrlhf,
title={How to Evaluate Reward Models for RLHF},
author={Evan Frick and Tianle Li and Connor Chen and Wei-Lin Chiang and Anastasios N. Angelopoulos and Jiantao Jiao and Banghua Zhu and Joseph E. Gonzalez and Ion Stoica},
year={2024},
eprint={2410.14872},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2410.14872},
}
```

# 概述
本数据集包含用于偏好代理评估（Preference Proxy Evaluations）的MMLU-Pro正确性偏好评估集。
其提示词采样自[MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro)。
本数据集仅用于基准测试与模型评估，不可用于模型训练。
[论文](https://arxiv.org/abs/2410.14872)
[代码](https://github.com/lmarena/PPE)

# 许可协议
用户提示词采用MIT许可协议授权，模型输出则受对应模型服务商的使用条款约束。

# 引用

@misc{frick2024evaluaterewardmodelsrlhf,
title={How to Evaluate Reward Models for RLHF},
author={Evan Frick and Tianle Li and Connor Chen and Wei-Lin Chiang and Anastasios N. Angelopoulos and Jiantao Jiao and Banghua Zhu and Joseph E. Gonzalez and Ion Stoica},
year={2024},
eprint={2410.14872},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2410.14872},
}

搜集汇总

背景与挑战

背景概述

该数据集基于MMLU-Pro采样，专为正确性偏好评估而设计，主要用于基准测试和评价，不适用于训练目的。