INSAIT-Institute/rit-mmlu
收藏资源简介：
MMLU多语言基准数据集，源自Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets项目（项目页面：https://ritranslation.insait.ai/，arXiv:2602.22207）。该数据集将INSAIT-Institute/multilingual-benchmarks集合中的每种语言数据集重组为一个统一的Hugging Face数据集存储库，每种语言作为一个独立的配置/子集。MMLU（Massive Multitask Language Understanding）是一个多项选择基准，涵盖57个学术和专业科目，常用于评估语言模型的世界知识和推理能力。RiTranslation论文提出了一种自动化高质量基准和数据集翻译框架，研究了包括Self-Check、Best-of-N、Universal Self-Improvement (USI)和T-RANK多轮排序方法在内的测试时计算策略，评估显示其提高了翻译质量和多语言LLM评估的可靠性。可用语言配置包括：乌克兰语(ukr)、斯洛伐克语(sk)、罗马尼亚语(ro)、爱沙尼亚语(et)、立陶宛语(lt)、希腊语(el)、土耳其语(tr)。数据集结构包含以下字段：问题(question)、主题(subject)、选项(choices)、答案(answer)。每种配置保留源数据集的分割（训练、验证、测试）和模式，便于加载和评估。该数据集旨在用于多语言基准评估和分析，使跨语言模型比较更加便捷，同时保持每种语言的隔离性。注意：这些是机器翻译的评估基准，结果解释需考虑语言特定歧义、文化背景和基准污染等因素。
The MMLU Multilingual benchmark dataset, released from the Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets project (project page: https://ritranslation.insait.ai/, arXiv:2602.22207). It reorganizes the per-language datasets from the INSAIT-Institute/multilingual-benchmarks collection into a single Hugging Face dataset repository, with one config/subset per language. MMLU (Massive Multitask Language Understanding) is a multiple-choice benchmark spanning 57 academic and professional subjects, commonly used to evaluate world knowledge and reasoning in language models. The RiTranslation paper introduces an automated framework for high-quality benchmark and dataset translation, studying test-time compute strategies including Self-Check, Best-of-N, Universal Self-Improvement (USI), and the T-RANK multi-round ranking method, with evaluations showing improved translation quality and more reliable multilingual LLM assessment. Available language configs: Ukrainian (ukr), Slovak (sk), Romanian (ro), Estonian (et), Lithuanian (lt), Greek (el), Turkish (tr). Dataset structure includes fields: question, subject, choices, answer. Each config preserves the split names and schema of the corresponding source dataset for easy loading and evaluation. Intended for multilingual benchmark evaluation and analysis, facilitating model comparison across languages while keeping each language isolated. Caveats: These are machine-translated evaluation benchmarks; results should be interpreted with care for language-specific ambiguity, cultural context, and benchmark contamination.
