sarvamai/mmlu-indic

Indic MMLU数据集是Massive Multitask Language Understanding (MMLU)基准的多语言版本，从英语翻译成10种印度语言。该版本仅包含开发和测试集的翻译。数据集涵盖的语言包括孟加拉语、古吉拉特语、印地语、卡纳达语、马拉地语、马拉雅拉姆语、奥里亚语、旁遮普语、泰米尔语和泰卢固语。每个示例是一个多项选择题，包含问题文本、四个可能的答案、正确答案的索引以及语言代码。数据集统计信息显示，验证集每种语言约有280个示例，测试集每种语言约有14,000个示例。数据集的使用方法是通过`datasets`库加载。已知限制包括技术术语的翻译可能不精确，某些主题（如美国法律）可能没有直接对应的概念，文化和教育系统的差异可能影响问题的相关性。数据集遵循与原始MMLU数据集相同的许可证。

The Indic MMLU dataset is a multilingual version of the MMLU benchmark, translated into 10 Indian languages including Bengali, Gujarati, Hindi, Kannada, Marathi, Malayalam, Oriya, Punjabi, Tamil, and Telugu. Each sample contains a question, four possible answers, the index of the correct answer, and the language code. The dataset is divided into validation and test sets, containing approximately 280 and 14k samples per language, respectively. The usage of the dataset is demonstrated through a Python code example. Known limitations include challenges in translating technical terminology, lack of direct equivalents for some subject concepts, and potential cultural and educational system differences affecting question relevance. The dataset follows the same license as the original MMLU dataset.