Name: MBZUAI-Paris/DarijaMMLU
Creator: MBZUAI-Paris
Published: 2024-09-27 08:09:38
License: 暂无描述

下载链接：

https://hf-mirror.com/datasets/MBZUAI-Paris/DarijaMMLU

下载链接

链接失效反馈

官方服务：

资源简介：

--- annotations_creators: - machine-generated language_creators: - machine-translated language: - ma license: - mit multilinguality: - monolingual size_categories: - 10K<n<100K source_datasets: - mmlu - arabicmmlu task_categories: - question-answering task_ids: - multiple-choice-qa dataset_info: - config_name: accounting features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 29650 num_examples: 74 - name: dev num_bytes: 1077 num_examples: 3 download_size: 20654 dataset_size: 30727 - config_name: arabic_language features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 242365 num_examples: 669 - name: dev num_bytes: 2656 num_examples: 9 download_size: 88282 dataset_size: 245021 - config_name: arabic_language_(general) features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 1465890 num_examples: 612 - name: dev num_bytes: 6338 num_examples: 3 download_size: 305164 dataset_size: 1472228 - config_name: arabic_language_(grammar) features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 132061 num_examples: 365 - name: dev num_bytes: 881 num_examples: 3 download_size: 29243 dataset_size: 132942 - config_name: biology features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 431076 num_examples: 1409 - name: dev num_bytes: 978 num_examples: 3 download_size: 160412 dataset_size: 432054 - config_name: civics features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 106902 num_examples: 323 - name: dev num_bytes: 1805 num_examples: 6 download_size: 45592 dataset_size: 108707 - config_name: computer_science features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 156535 num_examples: 542 - name: dev num_bytes: 3997 num_examples: 12 download_size: 60539 dataset_size: 160532 - config_name: driving_test features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 418951 num_examples: 1211 - name: dev num_bytes: 921 num_examples: 3 download_size: 146345 dataset_size: 419872 - config_name: economics features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 227729 num_examples: 584 - name: dev num_bytes: 2701 num_examples: 9 download_size: 86153 dataset_size: 230430 - config_name: general_knowledge features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 319620 num_examples: 1198 - name: dev num_bytes: 2984 num_examples: 9 download_size: 116762 dataset_size: 322604 - config_name: geography features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 414694 num_examples: 1367 - name: dev num_bytes: 2639 num_examples: 9 download_size: 133567 dataset_size: 417333 - config_name: global_facts features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 33514 num_examples: 100 - name: dev num_bytes: 1843 num_examples: 5 download_size: 20273 dataset_size: 35357 - config_name: high_school_european_history features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 386002 num_examples: 165 - name: dev num_bytes: 16803 num_examples: 5 download_size: 211022 dataset_size: 402805 - config_name: high_school_geography features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 78829 num_examples: 198 - name: dev num_bytes: 2428 num_examples: 5 download_size: 39743 dataset_size: 81257 - config_name: high_school_government_and_politics features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 117086 num_examples: 193 - name: dev num_bytes: 2953 num_examples: 5 download_size: 55211 dataset_size: 120039 - config_name: high_school_psychology features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 271010 num_examples: 545 - name: dev num_bytes: 2980 num_examples: 5 download_size: 117929 dataset_size: 273990 - config_name: high_school_statistics features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 173857 num_examples: 216 - name: dev num_bytes: 3905 num_examples: 5 download_size: 81341 dataset_size: 177762 - config_name: high_school_world_history features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 543247 num_examples: 237 - name: dev num_bytes: 7446 num_examples: 5 download_size: 266699 dataset_size: 550693 - config_name: history features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 325784 num_examples: 1065 - name: dev num_bytes: 2488 num_examples: 9 download_size: 119431 dataset_size: 328272 - config_name: human_aging features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 80694 num_examples: 223 - name: dev num_bytes: 1871 num_examples: 5 download_size: 43838 dataset_size: 82565 - config_name: international_law features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 84182 num_examples: 121 - name: dev num_bytes: 3617 num_examples: 5 download_size: 43915 dataset_size: 87799 - config_name: islamic_studies features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 621152 num_examples: 2210 - name: dev num_bytes: 3927 num_examples: 12 download_size: 200111 dataset_size: 625079 - config_name: jurisprudence features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 53464 num_examples: 108 - name: dev num_bytes: 2109 num_examples: 5 download_size: 33956 dataset_size: 55573 - config_name: law features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 143073 num_examples: 314 - name: dev num_bytes: 1849 num_examples: 3 download_size: 60957 dataset_size: 144922 - config_name: logical_fallacies features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 79322 num_examples: 163 - name: dev num_bytes: 2513 num_examples: 5 download_size: 40670 dataset_size: 81835 - config_name: management features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 32660 num_examples: 103 - name: dev num_bytes: 1488 num_examples: 5 download_size: 23335 dataset_size: 34148 - config_name: management_ar features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 23827 num_examples: 75 - name: dev num_bytes: 961 num_examples: 3 download_size: 17819 dataset_size: 24788 - config_name: marketing features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 99963 num_examples: 234 - name: dev num_bytes: 2523 num_examples: 5 download_size: 51795 dataset_size: 102486 - config_name: math features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 65505 num_examples: 409 - name: dev num_bytes: 525 num_examples: 3 download_size: 26903 dataset_size: 66030 - config_name: moral_disputes features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 175752 num_examples: 346 - name: dev num_bytes: 2973 num_examples: 5 download_size: 81475 dataset_size: 178725 - config_name: moral_scenarios features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 623891 num_examples: 895 - name: dev num_bytes: 3367 num_examples: 5 download_size: 132480 dataset_size: 627258 - config_name: natural_science features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 165706 num_examples: 578 - name: dev num_bytes: 1431 num_examples: 6 download_size: 63806 dataset_size: 167137 - config_name: nutrition features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 154223 num_examples: 306 - name: dev num_bytes: 3485 num_examples: 5 download_size: 74756 dataset_size: 157708 - config_name: philosophy features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 123687 num_examples: 311 - name: dev num_bytes: 1482 num_examples: 5 download_size: 61785 dataset_size: 125169 - config_name: philosophy_ar features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 17849 num_examples: 39 - name: dev num_bytes: 1312 num_examples: 3 download_size: 18413 dataset_size: 19161 - config_name: physics features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 89899 num_examples: 255 - name: dev num_bytes: 1112 num_examples: 3 download_size: 40890 dataset_size: 91011 - config_name: political_science features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 65495 num_examples: 210 - name: dev num_bytes: 902 num_examples: 3 download_size: 30993 dataset_size: 66397 - config_name: professional_law features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 2561850 num_examples: 1534 - name: dev num_bytes: 9007 num_examples: 5 download_size: 1153678 dataset_size: 2570857 - config_name: professional_psychology features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 358762 num_examples: 612 - name: dev num_bytes: 3529 num_examples: 5 download_size: 162315 dataset_size: 362291 - config_name: public_relations features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 47733 num_examples: 110 - name: dev num_bytes: 2395 num_examples: 5 download_size: 31046 dataset_size: 50128 - config_name: security_studies features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 301785 num_examples: 245 - name: dev num_bytes: 7606 num_examples: 5 download_size: 143714 dataset_size: 309391 - config_name: social_science features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 229262 num_examples: 946 - name: dev num_bytes: 1453 num_examples: 6 download_size: 77361 dataset_size: 230715 - config_name: sociology features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 105124 num_examples: 201 - name: dev num_bytes: 2630 num_examples: 5 download_size: 57628 dataset_size: 107754 - config_name: world_religions features: - name: question dtype: string - name: context dtype: string - name: choices sequence: string - name: answer dtype: int64 - name: subject dtype: string - name: subject_darija dtype: string - name: source dtype: string splits: - name: test num_bytes: 47534 num_examples: 171 - name: dev num_bytes: 1249 num_examples: 5 download_size: 27946 dataset_size: 48783 configs: - config_name: accounting data_files: - split: test path: accounting/test-* - split: dev path: accounting/dev-* - config_name: arabic_language data_files: - split: test path: arabic_language/test-* - split: dev path: arabic_language/dev-* - config_name: arabic_language_(general) data_files: - split: test path: arabic_language_(general)/test-* - split: dev path: arabic_language_(general)/dev-* - config_name: arabic_language_(grammar) data_files: - split: test path: arabic_language_(grammar)/test-* - split: dev path: arabic_language_(grammar)/dev-* - config_name: biology data_files: - split: test path: biology/test-* - split: dev path: biology/dev-* - config_name: civics data_files: - split: test path: civics/test-* - split: dev path: civics/dev-* - config_name: computer_science data_files: - split: test path: computer_science/test-* - split: dev path: computer_science/dev-* - config_name: driving_test data_files: - split: test path: driving_test/test-* - split: dev path: driving_test/dev-* - config_name: economics data_files: - split: test path: economics/test-* - split: dev path: economics/dev-* - config_name: general_knowledge data_files: - split: test path: general_knowledge/test-* - split: dev path: general_knowledge/dev-* - config_name: geography data_files: - split: test path: geography/test-* - split: dev path: geography/dev-* - config_name: global_facts data_files: - split: test path: global_facts/test-* - split: dev path: global_facts/dev-* - config_name: high_school_european_history data_files: - split: test path: high_school_european_history/test-* - split: dev path: high_school_european_history/dev-* - config_name: high_school_geography data_files: - split: test path: high_school_geography/test-* - split: dev path: high_school_geography/dev-* - config_name: high_school_government_and_politics data_files: - split: test path: high_school_government_and_politics/test-* - split: dev path: high_school_government_and_politics/dev-* - config_name: high_school_psychology data_files: - split: test path: high_school_psychology/test-* - split: dev path: high_school_psychology/dev-* - config_name: high_school_statistics data_files: - split: test path: high_school_statistics/test-* - split: dev path: high_school_statistics/dev-* - config_name: high_school_world_history data_files: - split: test path: high_school_world_history/test-* - split: dev path: high_school_world_history/dev-* - config_name: history data_files: - split: test path: history/test-* - split: dev path: history/dev-* - config_name: human_aging data_files: - split: test path: human_aging/test-* - split: dev path: human_aging/dev-* - config_name: international_law data_files: - split: test path: international_law/test-* - split: dev path: international_law/dev-* - config_name: islamic_studies data_files: - split: test path: islamic_studies/test-* - split: dev path: islamic_studies/dev-* - config_name: jurisprudence data_files: - split: test path: jurisprudence/test-* - split: dev path: jurisprudence/dev-* - config_name: law data_files: - split: test path: law/test-* - split: dev path: law/dev-* - config_name: logical_fallacies data_files: - split: test path: logical_fallacies/test-* - split: dev path: logical_fallacies/dev-* - config_name: management data_files: - split: test path: management/test-* - split: dev path: management/dev-* - config_name: management_ar data_files: - split: test path: management_ar/test-* - split: dev path: management_ar/dev-* - config_name: marketing data_files: - split: test path: marketing/test-* - split: dev path: marketing/dev-* - config_name: math data_files: - split: test path: math/test-* - split: dev path: math/dev-* - config_name: moral_disputes data_files: - split: test path: moral_disputes/test-* - split: dev path: moral_disputes/dev-* - config_name: moral_scenarios data_files: - split: test path: moral_scenarios/test-* - split: dev path: moral_scenarios/dev-* - config_name: natural_science data_files: - split: test path: natural_science/test-* - split: dev path: natural_science/dev-* - config_name: nutrition data_files: - split: test path: nutrition/test-* - split: dev path: nutrition/dev-* - config_name: philosophy data_files: - split: test path: philosophy/test-* - split: dev path: philosophy/dev-* - config_name: philosophy_ar data_files: - split: test path: philosophy_ar/test-* - split: dev path: philosophy_ar/dev-* - config_name: physics data_files: - split: test path: physics/test-* - split: dev path: physics/dev-* - config_name: political_science data_files: - split: test path: political_science/test-* - split: dev path: political_science/dev-* - config_name: professional_law data_files: - split: test path: professional_law/test-* - split: dev path: professional_law/dev-* - config_name: professional_psychology data_files: - split: test path: professional_psychology/test-* - split: dev path: professional_psychology/dev-* - config_name: public_relations data_files: - split: test path: public_relations/test-* - split: dev path: public_relations/dev-* - config_name: security_studies data_files: - split: test path: security_studies/test-* - split: dev path: security_studies/dev-* - config_name: social_science data_files: - split: test path: social_science/test-* - split: dev path: social_science/dev-* - config_name: sociology data_files: - split: test path: sociology/test-* - split: dev path: sociology/dev-* - config_name: world_religions data_files: - split: test path: world_religions/test-* - split: dev path: world_religions/dev-* --- # Dataset Card for DarijaMMLU ## Table of Contents - [Table of Contents](#table-of-contents) - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks](#supported-tasks) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Data Splits](#data-splits) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Source Data](#source-data) - [Annotations](#annotations) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) ## Dataset Description - **Homepage:** [https://hf.co/datasets/MBZUAI-Paris/DarijaMMLU](https://hf.co/datasets/MBZUAI-Paris/DarijaMMLU) - **Repository:** [https://github.com/MBZUAI-Paris/lm-evaluation-harness-Atlas-Chat](https://github.com/MBZUAI-Paris/lm-evaluation-harness-Atlas-Chat) - **Paper:** [More Information Needed]   ### Dataset Summary DarijaMMLU is an evaluation benchmark designed to assess large language models' (LLM) performance in Moroccan Darija, a variety of Arabic. It consists of 22,027 multiple-choice questions, translated from selected subsets of the Massive Multitask Language Understanding (MMLU) and ArabicMMLU benchmarks to measure model performance on 44 subjects in Darija. ### Supported Tasks - **Task Category:** Multiple-choice question answering - **Task:** Answering multiple-choice questions in Darija  ### Languages The dataset is available in Moroccan Arabic (Darija). ## Dataset Structure The dataset consists of 44 folders covering the 44 subjects included in the dataset. ### Data Instances Each data instance of each subject contains a multiple-choice question with 2 to 5 answer options. The structure includes: - **question**: The multiple-choice question in Darija. - **context**: Additional contextual information that may be useful for answering the question. - **choices**: A list of possible answer options. - **answer**: The correct answer to the question (0, 1, 2, 3, or 4). - **subject**: The subject category for the question. - **subject_darija**: The subject category in Darija. - **source**: The source from which the question was derived (either MMLU or ArabicMMLU). Example: ``` { "question": "اتخذ الرسول صلى الله عليه وسلم …….. بلاصة كيتجمع فيها مع صحابو.", "context": "", "choices": [ "غار حراء", "الجامع", "دار الأرقم", "مكة" ], "answer": 2, "subject": "islamic_studies", "subject_darija": "الدراسات الإسلامية", "source": "arabic_mmlu", "split": "test" } ``` ### Data Splits The dataset consists of two main splits: test and development. ## Dataset Creation ### Curation Rationale The dataset was created to address the need for high-quality, culturally relevant benchmarks for evaluating language models in Moroccan Darija. By translating and adapting established benchmarks, it allows for consistent evaluation across languages and domains. ### Source Data #### Initial Data Collection and Normalization The data was derived from two major benchmarks: - **Massive Multitask Language Understanding (MMLU)**: A large benchmark for multiple-choice question answering. - **ArabicMMLU**: An Arabic version of MMLU. The selected subsets were translated into Darija using Claude 3.5 Sonnet. #### Who are the source language producers? The source language producers are the original authors of MMLU and ArabicMMLU benchmarks. The translations were produced using machine translation with manual curation for quality control. ### Annotations #### Annotation process The dataset was created through a combination of machine translation and manual review to ensure linguistic accuracy and cultural appropriateness. #### Who are the annotators? The annotators include experts familiar with both Moroccan Darija.  ### Personal and Sensitive Information The dataset does not contain personal or sensitive information. ## Considerations for Using the Data ### Social Impact of Dataset This dataset promotes the development of language models capable of understanding and responding in Moroccan Darija, contributing to the advancement of NLP for underrepresented languages. ### Discussion of Biases The dataset excludes certain technical topics and culturally inappropriate questions to ensure relevance and accessibility in the Moroccan context. However, as the data was machine-translated and adapted, it may still contain linguistic biases inherent in the translation models used, namely Claude 3.5 Sonnet . ### Other Known Limitations - The dataset is limited to the topics and domains covered by MMLU and ArabicMMLU. ## Additional Information ### Dataset Curators - MBZUAI-Paris team ### Licensing Information - [MIT License](https://github.com/hendrycks/test/blob/master/LICENSE) ### Citation Information ``` @article{shang2024atlaschatadaptinglargelanguage, title={Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect}, author={Guokan Shang and Hadi Abdine and Yousef Khoubrane and Amr Mohamed and Yassine Abbahaddou and Sofiane Ennadir and Imane Momayiz and Xuguang Ren and Eric Moulines and Preslav Nakov and Michalis Vazirgiannis and Eric Xing}, year={2024}, eprint={2409.17912}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2409.17912}, } ``` ``` @article{hendryckstest2021, title={Measuring Massive Multitask Language Understanding}, author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt}, journal={Proceedings of the International Conference on Learning Representations (ICLR)}, year={2021} } ``` ``` @article{koto2024arabicmmlu, title={Arabicmmlu: Assessing massive multitask language understanding in arabic}, author={Koto, Fajri and Li, Haonan and Shatnawi, Sara and Doughman, Jad and Sadallah, Abdelrahman Boda and Alraeesi, Aisha and Almubarak, Khalid and Alyafeai, Zaid and Sengupta, Neha and Shehata, Shady and others}, journal={arXiv preprint arXiv:2402.12840}, year={2024} } ```

应用场景：