OpenLLM-Ro/ro_mmlu

---
license: cc-by-nc-4.0
language:
- ro
---

### Dataset Description

<!-- Provide a longer summary of what this dataset is. -->
[Measuring Massive Multitask Language Understanding (MMLU)](https://arxiv.org/abs/2009.03300) is a benchmark that measures a text model’s multitask accuracy.
The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.

Here we provide the Romanian translation of the MMLU from the paper *"Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback"* ([Lai et al., 2023](https://arxiv.org/abs/2307.16039)).
This dataset is used as a benchmark and is part of the evaluation protocol for Romanian LLMs proposed in *"Vorbeşti Româneşte?" A Recipe to Train Powerful Romanian LLMs with English Instructions* ([Masala et al., 2024](https://arxiv.org/abs/2406.18266))

## Citation

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

```bibtex
@article{dac2023okapi,
title={Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback},
author={Dac Lai, Viet and Van Nguyen, Chien and Ngo, Nghia Trung and Nguyen, Thuat and Dernoncourt, Franck and Rossi, Ryan A and Nguyen, Thien Huu},
journal={arXiv e-prints},
pages={arXiv--2307},
year={2023}
}
```

```bibtex
@article{hendryckstest2021,
title={Measuring Massive Multitask Language Understanding},
author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},
journal={Proceedings of the International Conference on Learning Representations (ICLR)},
year={2021}
}
```

```bibtext
@article{masala2024vorbecstiromanecsterecipetrain,
title={"Vorbe\c{s}ti Rom\^ane\c{s}te?" A Recipe to Train Powerful Romanian LLMs with English Instructions},
author={Mihai Masala and Denis C. Ilie-Ablachim and Alexandru Dima and Dragos Corlatescu and Miruna Zavelca and Ovio Olaru and Simina Terian and Andrei Terian and Marius Leordeanu and Horia Velicu and Marius Popescu and Mihai Dascalu and Traian Rebedea},
year={2024},
eprint={2406.18266},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

许可证：知识共享署名-非商业性使用4.0国际许可协议（CC BY-NC 4.0）
语言：罗马尼亚语

### 数据集描述

大规模多任务语言理解（Measuring Massive Multitask Language Understanding，MMLU）是一项用于评估文本模型多任务准确率的基准测试集。该测试涵盖57项任务，包括初等数学、美国历史、计算机科学、法学等多个领域。

本数据集提供了来自论文《Okapi：结合人类反馈强化学习的多语言指令微调大语言模型（Large Language Model，LLM）》（Lai等，2023）的MMLU罗马尼亚语译版。

本数据集作为基准测试集，被应用于论文《"您会说罗马尼亚语吗？"：使用英语指令训练高性能罗马尼亚语大语言模型（Large Language Model，LLM）》（Masala等，2024）中提出的罗马尼亚语大语言模型测评方案。

## 引用

bibtex
@article{dac2023okapi,
title={Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback},
author={Dac Lai, Viet and Van Nguyen, Chien and Ngo, Nghia Trung and Nguyen, Thuat and Dernoncourt, Franck and Rossi, Ryan A and Nguyen, Thien Huu},
journal={arXiv e-prints},
pages={arXiv--2307},
year={2023}
}

bibtex
@article{hendryckstest2021,
title={Measuring Massive Multitask Language Understanding},
author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},
journal={Proceedings of the International Conference on Learning Representations (ICLR)},
year={2021}
}

bibtex
@article{masala2024vorbecstiromanecsterecipetrain,
title={"Vorbec{s}ti Rom^anec{s}te?" A Recipe to Train Powerful Romanian LLMs with English Instructions},
author={Mihai Masala and Denis C. Ilie-Ablachim and Alexandru Dima and Dragos Corlatescu and Miruna Zavelca and Ovio Olaru and Simina Terian and Andrei Terian and Marius Leordeanu and Horia Velicu and Marius Popescu and Mihai Dascalu and Traian Rebedea},
year={2024},
eprint={2406.18266},
archivePrefix={arXiv},
primaryClass={cs.CL}
}