jon-tow/okapi_mmlu

---
language:
- ar
- bn
- ca
- da
- de
- es
- eu
- fr
- gu
- hi
- hr
- hu
- hy
- id
- it
- kn
- ml
- mr
- ne
- nl
- pt
- ro
- ru
- sk
- sr
- sv
- ta
- te
- uk
- vi
license: cc-by-nc-4.0
---

# okapi_mmlu

<!-- Provide a quick summary of the dataset. -->
Multilingual translation of [Measuring Massive Multitask Language Understanding (MMLU)](https://arxiv.org/abs/2009.03300).

## Dataset Details

### Dataset Description

<!-- Provide a longer summary of what this dataset is. -->
MMLU is a benchmark that measured a text model’s multitask accuracy.
The test covers 57 tasks including elementary mathematics, US history, computer
science, law, and more. To attain high accuracy on this test, models must possess
extensive world knowledge and problem solving ability. By comprehensively evaluating the breadth and depth of a model’s academic and professional understanding, MMLU can be used to analyze models across many tasks and to identify important shortcomings.

- **Curated by:** Dac Lai, Viet and Van Nguyen, Chien and Ngo, Nghia Trung and Nguyen, Thuat and Dernoncourt, Franck and Rossi, Ryan A and Nguyen, Thien Huu
- **License:** The datasets are CC BY NC 4.0 (allowing only non-commercial use).

### Dataset Sources

<!-- Provide the basic links for the dataset. -->

- **Repository:** http://nlp.uoregon.edu/download/okapi-eval/datasets/
- **Paper:** Okapi ([Lai et al., 2023](https://arxiv.org/abs/2307.16039))

## Citation

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

```bibtex
@article{dac2023okapi,
title={Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback},
author={Dac Lai, Viet and Van Nguyen, Chien and Ngo, Nghia Trung and Nguyen, Thuat and Dernoncourt, Franck and Rossi, Ryan A and Nguyen, Thien Huu},
journal={arXiv e-prints},
pages={arXiv--2307},
year={2023}
}
```

```bibtex
@article{hendryckstest2021,
title={Measuring Massive Multitask Language Understanding},
author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},
journal={Proceedings of the International Conference on Learning Representations (ICLR)},
year={2021}
}
```

原始信息汇总

okapi_mmlu

数据集详情

数据集描述

MMLU是一个用于评估文本模型多任务准确性的基准测试。该测试涵盖57个任务，包括基础数学、美国历史、计算机科学、法律等多个领域。为了在这个测试中获得高准确性，模型必须具备广泛的世界知识和问题解决能力。通过全面评估模型在学术和专业领域的广度和深度理解，MMLU可用于分析模型在多个任务中的表现，并识别重要的不足之处。

• 由以下人员策划： Dac Lai, Viet 和 Van Nguyen, Chien 和 Ngo, Nghia Trung 和 Nguyen, Thuat 和 Dernoncourt, Franck 和 Rossi, Ryan A 和 Nguyen, Thien Huu
• 许可证： 数据集采用CC BY NC 4.0许可证（仅允许非商业使用）。

数据集来源

• 仓库： http://nlp.uoregon.edu/download/okapi-eval/datasets/
• 论文： Okapi (Lai et al., 2023)

引用

bibtex @article{dac2023okapi, title={Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback}, author={Dac Lai, Viet and Van Nguyen, Chien and Ngo, Nghia Trung and Nguyen, Thuat and Dernoncourt, Franck and Rossi, Ryan A and Nguyen, Thien Huu}, journal={arXiv e-prints}, pages={arXiv--2307}, year={2023} }

bibtex @article{hendryckstest2021, title={Measuring Massive Multitask Language Understanding}, author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt}, journal={Proceedings of the International Conference on Learning Representations (ICLR)}, year={2021} }