Dialectal-Arabic-MMLU

# DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models ## Dataset Summary Dialectal-Arabic-MMLU is a large-scale, human-translated for MMLU. We extend MMLU-Redux into 5 major dialects: Syrian, Egyptian, Emirati, Saudi, and Moroccan. This data covers 21K QA pairs across 32 academic and professional domains. More details, please check our paper on [DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models](http://www.lrec-conf.org/proceedings/lrec2026/pdf/2026.lrec2026-1.251.pdf) ## Supported Languages and Dialects This data covers English and MSA, as well as 5 dialects: Syrian, Egyptian, Emirati, Saudi, and Moroccan. ## Dataset Structure This dataset is derived from **MMLU-Redux-v2** and follows a multiple-choice question-answering format with four candidate `choices`. The correct label is stored in the `answer` field. Each question is assigned a unique `qid` (ranging from 0–99) within its respective `domain`. These IDs are parallel across all dialects; for example, `qid` 8 in the `high_school_us_history` domain refers to the same underlying question across every dialectal variation. ### Domains We cover the following domains: High School US History, High School World History, International Law, Moral Scenarios, Philosophy, Prehistory, Professional Law, World Religions, Abstract Algebra, Anatomy, Astronomy, College Computer Science, Conceptual Physics, Elementary Mathematics, High School Chemistry, High School Geography, High School Macroeconomics, High School Psychology, Professional Psychology, Public Relations, Security Studies, Sociology, US Foreign Policy, Business Ethics, Clinical Knowledge, College Medicine, Global Facts, Human Aging, Management, Marketing, Nutrition, Virology. ### Data Fields * `dialect`: The specific dialect or language variant. * `qid`: The question id which is parallel across all dialects. * `question`: The multiple-choice question text. * `choices`: A list of strings containing four answer options. * `answer`: The index of the correct option (0-3). * `domain`: The subject domain of the question. ## Citation ```bibtex @inproceedings{altakrori-etal-2026-dialectalarabicmmlu, title = {{DialectalArabicMMLU}: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models}, author = {Altakrori, Malik H. and Habash, Nizar and Lynn, Teresa and Samih, Younes and Freihat, Abed Alhakim and Chirkunov, Kirill and AbuOdeh, Muhammed and Florian, Radu and Nakov, Preslav and Aji, Alham Fikri}, booktitle = {Proc. of the 15th Language Resources and Evaluation Conference (LREC)}, month = may, year = {2026}, pages = {3199--3219}, address = {Palma, Mallorca, Spain}, publisher = {European Language Resources Association (ELRA)}, doi = {10.63317/3cy68duew55b}, url= {https://doi.org/10.63317/3cy68duew55b}, } ```