five

NLPCoreTeam/mmlu_ru

收藏
Hugging Face2023-06-28 更新2024-03-04 收录
下载链接：
https://hf-mirror.com/datasets/NLPCoreTeam/mmlu_ru
下载链接
链接失效反馈
官方服务：
资源简介：
--- pretty_name: MMLU RU/EN language: - ru - en size_categories: - 10K<n<100K task_categories: - question-answering - multiple-choice task_ids: - multiple-choice-qa dataset_info: - config_name: abstract_algebra features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 2182 num_examples: 5 - name: val num_bytes: 5220 num_examples: 11 - name: test num_bytes: 50926 num_examples: 100 download_size: 5548198 dataset_size: 58328 - config_name: anatomy features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 2482 num_examples: 5 - name: val num_bytes: 8448 num_examples: 14 - name: test num_bytes: 91387 num_examples: 135 download_size: 5548198 dataset_size: 102317 - config_name: astronomy features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 6049 num_examples: 5 - name: val num_bytes: 14187 num_examples: 16 - name: test num_bytes: 130167 num_examples: 152 download_size: 5548198 dataset_size: 150403 - config_name: business_ethics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 6197 num_examples: 5 - name: val num_bytes: 8963 num_examples: 11 - name: test num_bytes: 96566 num_examples: 100 download_size: 5548198 dataset_size: 111726 - config_name: clinical_knowledge features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3236 num_examples: 5 - name: val num_bytes: 18684 num_examples: 29 - name: test num_bytes: 178043 num_examples: 265 download_size: 5548198 dataset_size: 199963 - config_name: college_biology features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 4232 num_examples: 5 - name: val num_bytes: 13521 num_examples: 16 - name: test num_bytes: 139322 num_examples: 144 download_size: 5548198 dataset_size: 157075 - config_name: college_chemistry features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3533 num_examples: 5 - name: val num_bytes: 6157 num_examples: 8 - name: test num_bytes: 65540 num_examples: 100 download_size: 5548198 dataset_size: 75230 - config_name: college_computer_science features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 7513 num_examples: 5 - name: val num_bytes: 13341 num_examples: 11 - name: test num_bytes: 120578 num_examples: 100 download_size: 5548198 dataset_size: 141432 - config_name: college_mathematics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3841 num_examples: 5 - name: val num_bytes: 6835 num_examples: 11 - name: test num_bytes: 65110 num_examples: 100 download_size: 5548198 dataset_size: 75786 - config_name: college_medicine features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 4659 num_examples: 5 - name: val num_bytes: 22116 num_examples: 22 - name: test num_bytes: 235856 num_examples: 173 download_size: 5548198 dataset_size: 262631 - config_name: college_physics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3740 num_examples: 5 - name: val num_bytes: 9491 num_examples: 11 - name: test num_bytes: 81480 num_examples: 102 download_size: 5548198 dataset_size: 94711 - config_name: computer_security features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3150 num_examples: 5 - name: val num_bytes: 12859 num_examples: 11 - name: test num_bytes: 77969 num_examples: 100 download_size: 5548198 dataset_size: 93978 - config_name: conceptual_physics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 2611 num_examples: 5 - name: val num_bytes: 12480 num_examples: 26 - name: test num_bytes: 112243 num_examples: 235 download_size: 5548198 dataset_size: 127334 - config_name: econometrics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 4548 num_examples: 5 - name: val num_bytes: 13874 num_examples: 12 - name: test num_bytes: 128633 num_examples: 114 download_size: 5548198 dataset_size: 147055 - config_name: electrical_engineering features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 2598 num_examples: 5 - name: val num_bytes: 8003 num_examples: 16 - name: test num_bytes: 70846 num_examples: 145 download_size: 5548198 dataset_size: 81447 - config_name: elementary_mathematics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3760 num_examples: 5 - name: val num_bytes: 23416 num_examples: 41 - name: test num_bytes: 181090 num_examples: 378 download_size: 5548198 dataset_size: 208266 - config_name: formal_logic features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 4715 num_examples: 5 - name: val num_bytes: 17099 num_examples: 14 - name: test num_bytes: 133930 num_examples: 126 download_size: 5548198 dataset_size: 155744 - config_name: global_facts features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3450 num_examples: 5 - name: val num_bytes: 4971 num_examples: 10 - name: test num_bytes: 51481 num_examples: 100 download_size: 5548198 dataset_size: 59902 - config_name: high_school_biology features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 4759 num_examples: 5 - name: val num_bytes: 30807 num_examples: 32 - name: test num_bytes: 310356 num_examples: 310 download_size: 5548198 dataset_size: 345922 - config_name: high_school_chemistry features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3204 num_examples: 5 - name: val num_bytes: 18948 num_examples: 22 - name: test num_bytes: 158246 num_examples: 203 download_size: 5548198 dataset_size: 180398 - config_name: high_school_computer_science features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 7933 num_examples: 5 - name: val num_bytes: 9612 num_examples: 9 - name: test num_bytes: 126403 num_examples: 100 download_size: 5548198 dataset_size: 143948 - config_name: high_school_european_history features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 32447 num_examples: 5 - name: val num_bytes: 83098 num_examples: 18 - name: test num_bytes: 754136 num_examples: 165 download_size: 5548198 dataset_size: 869681 - config_name: high_school_geography features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 4131 num_examples: 5 - name: val num_bytes: 12467 num_examples: 22 - name: test num_bytes: 119021 num_examples: 198 download_size: 5548198 dataset_size: 135619 - config_name: high_school_government_and_politics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 5188 num_examples: 5 - name: val num_bytes: 20564 num_examples: 21 - name: test num_bytes: 194050 num_examples: 193 download_size: 5548198 dataset_size: 219802 - config_name: high_school_macroeconomics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3942 num_examples: 5 - name: val num_bytes: 37243 num_examples: 43 - name: test num_bytes: 340699 num_examples: 390 download_size: 5548198 dataset_size: 381884 - config_name: high_school_mathematics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3244 num_examples: 5 - name: val num_bytes: 14758 num_examples: 29 - name: test num_bytes: 140257 num_examples: 270 download_size: 5548198 dataset_size: 158259 - config_name: high_school_microeconomics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3503 num_examples: 5 - name: val num_bytes: 22212 num_examples: 26 - name: test num_bytes: 219097 num_examples: 238 download_size: 5548198 dataset_size: 244812 - config_name: high_school_physics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3905 num_examples: 5 - name: val num_bytes: 18535 num_examples: 17 - name: test num_bytes: 162917 num_examples: 151 download_size: 5548198 dataset_size: 185357 - config_name: high_school_psychology features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 5207 num_examples: 5 - name: val num_bytes: 49277 num_examples: 60 - name: test num_bytes: 455603 num_examples: 545 download_size: 5548198 dataset_size: 510087 - config_name: high_school_statistics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 6823 num_examples: 5 - name: val num_bytes: 28020 num_examples: 23 - name: test num_bytes: 312578 num_examples: 216 download_size: 5548198 dataset_size: 347421 - config_name: high_school_us_history features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 25578 num_examples: 5 - name: val num_bytes: 91278 num_examples: 22 - name: test num_bytes: 842680 num_examples: 204 download_size: 5548198 dataset_size: 959536 - config_name: high_school_world_history features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 13893 num_examples: 5 - name: val num_bytes: 129121 num_examples: 26 - name: test num_bytes: 1068018 num_examples: 237 download_size: 5548198 dataset_size: 1211032 - config_name: human_aging features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 2820 num_examples: 5 - name: val num_bytes: 13442 num_examples: 23 - name: test num_bytes: 132242 num_examples: 223 download_size: 5548198 dataset_size: 148504 - config_name: human_sexuality features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3072 num_examples: 5 - name: val num_bytes: 6699 num_examples: 12 - name: test num_bytes: 90007 num_examples: 131 download_size: 5548198 dataset_size: 99778 - config_name: international_law features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 6880 num_examples: 5 - name: val num_bytes: 19166 num_examples: 13 - name: test num_bytes: 157259 num_examples: 121 download_size: 5548198 dataset_size: 183305 - config_name: jurisprudence features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3568 num_examples: 5 - name: val num_bytes: 10638 num_examples: 11 - name: test num_bytes: 97121 num_examples: 108 download_size: 5548198 dataset_size: 111327 - config_name: logical_fallacies features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 4526 num_examples: 5 - name: val num_bytes: 14547 num_examples: 18 - name: test num_bytes: 144501 num_examples: 163 download_size: 5548198 dataset_size: 163574 - config_name: machine_learning features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 6966 num_examples: 5 - name: val num_bytes: 8986 num_examples: 11 - name: test num_bytes: 95571 num_examples: 112 download_size: 5548198 dataset_size: 111523 - config_name: management features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 2427 num_examples: 5 - name: val num_bytes: 5210 num_examples: 11 - name: test num_bytes: 57201 num_examples: 103 download_size: 5548198 dataset_size: 64838 - config_name: marketing features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 4514 num_examples: 5 - name: val num_bytes: 20832 num_examples: 25 - name: test num_bytes: 181786 num_examples: 234 download_size: 5548198 dataset_size: 207132 - config_name: medical_genetics features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3226 num_examples: 5 - name: val num_bytes: 8214 num_examples: 11 - name: test num_bytes: 57064 num_examples: 100 download_size: 5548198 dataset_size: 68504 - config_name: miscellaneous features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 1782 num_examples: 5 - name: val num_bytes: 39225 num_examples: 86 - name: test num_bytes: 407209 num_examples: 783 download_size: 5548198 dataset_size: 448216 - config_name: moral_disputes features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 4910 num_examples: 5 - name: val num_bytes: 36026 num_examples: 38 - name: test num_bytes: 313611 num_examples: 346 download_size: 5548198 dataset_size: 354547 - config_name: moral_scenarios features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 6175 num_examples: 5 - name: val num_bytes: 129062 num_examples: 100 - name: test num_bytes: 1137631 num_examples: 895 download_size: 5548198 dataset_size: 1272868 - config_name: nutrition features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 6030 num_examples: 5 - name: val num_bytes: 24210 num_examples: 33 - name: test num_bytes: 266173 num_examples: 306 download_size: 5548198 dataset_size: 296413 - config_name: philosophy features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 2631 num_examples: 5 - name: val num_bytes: 25751 num_examples: 34 - name: test num_bytes: 227086 num_examples: 311 download_size: 5548198 dataset_size: 255468 - config_name: prehistory features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 5394 num_examples: 5 - name: val num_bytes: 28687 num_examples: 35 - name: test num_bytes: 251723 num_examples: 324 download_size: 5548198 dataset_size: 285804 - config_name: professional_accounting features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 6277 num_examples: 5 - name: val num_bytes: 40914 num_examples: 31 - name: test num_bytes: 364528 num_examples: 282 download_size: 5548198 dataset_size: 411719 - config_name: professional_law features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 19120 num_examples: 5 - name: val num_bytes: 589307 num_examples: 170 - name: test num_bytes: 5479411 num_examples: 1534 download_size: 5548198 dataset_size: 6087838 - config_name: professional_medicine features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 10901 num_examples: 5 - name: val num_bytes: 69703 num_examples: 31 - name: test num_bytes: 633483 num_examples: 272 download_size: 5548198 dataset_size: 714087 - config_name: professional_psychology features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 6430 num_examples: 5 - name: val num_bytes: 82745 num_examples: 69 - name: test num_bytes: 648634 num_examples: 612 download_size: 5548198 dataset_size: 737809 - config_name: public_relations features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 4384 num_examples: 5 - name: val num_bytes: 13108 num_examples: 12 - name: test num_bytes: 82403 num_examples: 110 download_size: 5548198 dataset_size: 99895 - config_name: security_studies features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 16064 num_examples: 5 - name: val num_bytes: 67877 num_examples: 27 - name: test num_bytes: 611059 num_examples: 245 download_size: 5548198 dataset_size: 695000 - config_name: sociology features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 4693 num_examples: 5 - name: val num_bytes: 20654 num_examples: 22 - name: test num_bytes: 191420 num_examples: 201 download_size: 5548198 dataset_size: 216767 - config_name: us_foreign_policy features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 4781 num_examples: 5 - name: val num_bytes: 9171 num_examples: 11 - name: test num_bytes: 81649 num_examples: 100 download_size: 5548198 dataset_size: 95601 - config_name: virology features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 3063 num_examples: 5 - name: val num_bytes: 15618 num_examples: 18 - name: test num_bytes: 111027 num_examples: 166 download_size: 5548198 dataset_size: 129708 - config_name: world_religions features: - name: question_en dtype: string - name: choices_en sequence: string - name: answer dtype: class_label: names: '0': A '1': B '2': C '3': D - name: question_ru dtype: string - name: choices_ru sequence: string splits: - name: dev num_bytes: 1691 num_examples: 5 - name: val num_bytes: 7052 num_examples: 19 - name: test num_bytes: 65559 num_examples: 171 download_size: 5548198 dataset_size: 74302 --- # MMLU in Russian (Massive Multitask Language Understanding) ## Overview of the Dataset MMLU dataset for EN/RU, without auxiliary train. The dataset contains `dev`/`val`/`test` splits for both, English and Russian languages. Note it doesn't include `auxiliary_train` split, which wasn't translated. Totally the dataset has ~16k samples per language: 285 `dev`, 1531 `val`, 14042 `test`. ## Description of original MMLU MMLU dataset covers 57 different tasks. Each task requires to choose the right answer out of four options for a given question. Paper "Measuring Massive Multitask Language Understanding": https://arxiv.org/abs/2009.03300v3. It is also known as the "hendrycks_test". ## Dataset Creation The translation was made via Yandex.Translate API. There are some translation mistakes, especially observed with terms and formulas, no fixes were applied. Initial dataset was taken from: https://people.eecs.berkeley.edu/~hendrycks/data.tar. ## Sample example ``` { "question_en": "Why doesn't Venus have seasons like Mars and Earth do?", "choices_en": [ "Its rotation axis is nearly perpendicular to the plane of the Solar System.", "It does not have an ozone layer.", "It does not rotate fast enough.", "It is too close to the Sun." ], "answer": 0, "question_ru": "Почему на Венере нет времен года, как на Марсе и Земле?", "choices_ru": [ "Ось его вращения почти перпендикулярна плоскости Солнечной системы.", "У него нет озонового слоя.", "Он вращается недостаточно быстро.", "Это слишком близко к Солнцу." ] } ``` ## Usage To merge all subsets into dataframe per split: ```python from collections import defaultdict import datasets import pandas as pd subjects = ["abstract_algebra", "anatomy", "astronomy", "business_ethics", "clinical_knowledge", "college_biology", "college_chemistry", "college_computer_science", "college_mathematics", "college_medicine", "college_physics", "computer_security", "conceptual_physics", "econometrics", "electrical_engineering", "elementary_mathematics", "formal_logic", "global_facts", "high_school_biology", "high_school_chemistry", "high_school_computer_science", "high_school_european_history", "high_school_geography", "high_school_government_and_politics", "high_school_macroeconomics", "high_school_mathematics", "high_school_microeconomics", "high_school_physics", "high_school_psychology", "high_school_statistics", "high_school_us_history", "high_school_world_history", "human_aging", "human_sexuality", "international_law", "jurisprudence", "logical_fallacies", "machine_learning", "management", "marketing", "medical_genetics", "miscellaneous", "moral_disputes", "moral_scenarios", "nutrition", "philosophy", "prehistory", "professional_accounting", "professional_law", "professional_medicine", "professional_psychology", "public_relations", "security_studies", "sociology", "us_foreign_policy", "virology", "world_religions"] splits = ["dev", "val", "test"] all_datasets = {x: datasets.load_dataset("NLPCoreTeam/mmlu_ru", name=x) for x in subjects} res = defaultdict(list) for subject in subjects: for split in splits: dataset = all_datasets[subject][split] df = dataset.to_pandas() int2str = dataset.features['answer'].int2str df['answer'] = df['answer'].map(int2str) df.insert(loc=0, column='subject_en', value=subject) res[split].append(df) res = {k: pd.concat(v) for k, v in res.items()} df_dev = res['dev'] df_val = res['val'] df_test = res['test'] ``` ## Evaluation This dataset is intended to evaluate LLMs with few-shot/zero-shot setup. Evaluation code: https://github.com/NLP-Core-Team/mmlu_ru Also resources might be helpful: 1. https://github.com/hendrycks/test 1. https://github.com/openai/evals/blob/main/examples/mmlu.ipynb 1. https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/hendrycks_test.py ## Contributions Dataset added by NLP core team RnD [Telegram channel](https://t.me/nlpcoreteam)
提供机构：
NLPCoreTeam
原始信息汇总

MMLU RU/EN 数据集概述

基本信息

  • 名称: MMLU RU/EN
  • 语言: 俄语 (ru), 英语 (en)
  • 大小: 10K<n<100K
  • 任务类型: 问答, 多选题
  • 任务ID: multiple-choice-qa

数据集结构

数据集包含多个子配置，每个配置对应不同的学科领域，具有以下共同特征：

  • question_en: 英文问题，数据类型为字符串。
  • choices_en: 英文选项，数据类型为序列字符串。
  • answer: 答案，数据类型为分类标签，选项为A, B, C, D。
  • question_ru: 俄文问题，数据类型为字符串。
  • choices_ru: 俄文选项，数据类型为序列字符串。

数据集拆分

每个子配置数据集被拆分为开发集(dev), 验证集(val), 测试集(test)，具体信息如下：

开发集(dev)

  • 示例数量: 5
  • 字节数: 不同学科领域字节数不同

验证集(val)

  • 示例数量: 不同学科领域示例数量不同
  • 字节数: 不同学科领域字节数不同

测试集(test)

  • 示例数量: 不同学科领域示例数量不同
  • 字节数: 不同学科领域字节数不同

数据集大小

  • 下载大小: 5548198字节
  • 数据集大小: 不同学科领域数据集大小不同

学科领域配置列表

  1. abstract_algebra

    • 开发集字节数: 2182
    • 验证集字节数: 5220
    • 测试集字节数: 50926
    • 数据集大小: 58328

  2. anatomy

    • 开发集字节数: 2482
    • 验证集字节数: 8448
    • 测试集字节数: 91387
    • 数据集大小: 102317

  3. astronomy

    • 开发集字节数: 6049
    • 验证集字节数: 14187
    • 测试集字节数: 130167
    • 数据集大小: 150403

  4. business_ethics

    • 开发集字节数: 6197
    • 验证集字节数: 8963
    • 测试集字节数: 96566
    • 数据集大小: 111726

  5. clinical_knowledge

    • 开发集字节数: 3236
    • 验证集字节数: 18684
    • 测试集字节数: 178043
    • 数据集大小: 199963

  6. college_biology

    • 开发集字节数: 4232
    • 验证集字节数: 13521
    • 测试集字节数: 139322
    • 数据集大小: 157075

  7. college_chemistry

    • 开发集字节数: 3533
    • 验证集字节数: 6157
    • 测试集字节数: 65540
    • 数据集大小: 75230

  8. college_computer_science

    • 开发集字节数: 7513
    • 验证集字节数: 13341
    • 测试集字节数: 120578
    • 数据集大小: 141432

  9. college_mathematics

    • 开发集字节数: 3841
    • 验证集字节数: 6835
    • 测试集字节数: 65110
    • 数据集大小: 75786

  10. college_medicine

    • 开发集字节数: 4659
    • 验证集字节数: 22116
    • 测试集字节数: 235856
    • 数据集大小: 262631

  11. college_physics

    • 开发集字节数: 3740
    • 验证集字节数: 9491
    • 测试集字节数: 81480
    • 数据集大小: 94711

  12. computer_security

    • 开发集字节数: 3150
    • 验证集字节数: 12859
    • 测试集字节数: 77969
    • 数据集大小: 93978

  13. conceptual_physics

    • 开发集字节数: 2611
    • 验证集字节数: 12480
    • 测试集字节数: 112243
    • 数据集大小: 127334

  14. econometrics

    • 开发集字节数: 4548
    • 验证集字节数: 13874
    • 测试集字节数: 128633
    • 数据集大小: 147055

  15. electrical_engineering

    • 开发集字节数: 2598
    • 验证集字节数: 8003
    • 测试集字节数: 70846
    • 数据集大小: 81447

  16. elementary_mathematics

    • 开发集字节数: 3760
    • 验证集字节数: 23416
    • 测试集字节数: 181090
    • 数据集大小: 208266

  17. formal_logic

    • 开发集字节数: 4715
    • 验证集字节数: 17099
    • 测试集字节数: 133930
    • 数据集大小: 155744

  18. global_facts

    • 开发集字节数: 3450
    • 验证集字节数: 4971
    • 测试集字节数: 51481
    • 数据集大小: 59902

  19. high_school_biology

    • 开发集字节数: 4759
    • 验证集字节数: 30807
    • 测试集字节数: 310356
    • 数据集大小: 345922

  20. high_school_chemistry

    • 开发集字节数: 3204
    • 验证集字节数: 18948
    • 测试集字节数: 158246
    • 数据集大小: 180398

  21. high_school_computer_science

    • 开发集字节数: 7933
    • 验证集字节数: 9612
    • 测试集字节数: 126403
    • 数据集大小: 143948

  22. high_school_european_history

    • 开发集字节数: 32447
    • 验证集字节数: 83098
    • 测试集字节数: 754136
    • 数据集大小: 869681

  23. high_school_geography

    • 开发集字节数: 4131
    • 验证集字节数: 12467
    • 测试集字节数: 119021
    • 数据集大小: 135619

  24. high_school_government_and_politics

    • 开发集字节数: 5188
    • 验证集字节数: 20564
    • 测试集字节数: 194050
    • 数据集大小: 219802

  25. high_school_macroeconomics

    • 开发集字节数: 3942
    • 验证集字节数: 37243
    • 测试集字节数: 340699
    • 数据集大小: 381884

  26. high_school_mathematics

    • 开发集字节数: 3244
    • 验证集字节数: 14758
    • 测试集字节数: 140257
    • 数据集大小: 158259

  27. high_school_microeconomics

    • 开发集字节数: 3503
    • 验证集字节数: 22212
    • 测试集字节数: 219097
    • 数据集大小: 244812

  28. high_school_physics

    • 开发集字节数: 3905
    • 验证集字节数: 18535
    • 测试集字节数: 162917
    • 数据集大小: 185357

  29. high_school_psychology

    • 开发集字节数: 5207
    • 验证集字节数: 49277
    • 测试集字节数: 455603
    • 数据集大小: 510087

  30. high_school_statistics

    • 开发集字节数: 6823
    • 验证集字节数: 28020
    • 测试集字节数: 312578
    • 数据集大小: 347421

  31. high_school_us_history

    • 开发集字节数: 25578
    • 验证集字节数: 91278
    • 测试集字节数: 842680
    • 数据集大小: 959536

  32. high_school_world_history

    • 开发集字节数: 13893
    • 验证集字节数: 129121
    • 测试集字节数: 1068018
    • 数据集大小: 1211032

  33. human_aging

    • 开发集字节数: 2820
    • 验证集字节数: 13442
    • 测试集字节数: 132242
    • 数据集大小: 148504

  34. human_sexuality

    • 开发集字节数: 3072
    • 验证集字节数: 6699
    • 测试集字节数: 90007
    • 数据集大小: 99778

  35. international_law

    • 开发集字节数: 6880
    • 验证集字节数: 19166
    • 测试集字节数: 157259
    • 数据集大小: 173305
搜集汇总
数据集介绍
main_image_url
构建方式
在自然语言处理与多语言知识评估的交叉领域中，MMLU（Massive Multitask Language Understanding）基准测试因其覆盖广泛学科而备受关注。NLPCoreTeam/mmlu_ru数据集正是基于这一经典基准，通过专业翻译与本地化流程构建而成。该数据集保留了MMLU原有的57个学科配置，涵盖从抽象代数到国际法等多元领域，每个配置均包含英文原版问题（question_en）、四个选项（choices_en）及标准答案（answer），并同步提供俄语版本的问题（question_ru）与选项（choices_ru）。数据划分遵循dev、val、test三部分，其中dev集统一为5个样本，val与test集样本量因学科而异，整体规模介于10K至100K之间，确保了评估的统计有效性。
特点
该数据集最显著的特点在于其双语对齐的精细结构，每个样本均包含英文与俄文两种语言形态的完整问答对，为跨语言模型评估提供了天然对照。学科覆盖从高中数学、物理等基础科目到商业伦理、国际法等专业领域，共计57个细分配置，全面反映了模型在多学科知识与推理能力上的表现。数据规模虽中等，但每个学科的test集均包含100至数百个样本，且答案采用A、B、C、D四选一的标准化标签体系，便于自动化评测。此外，所有学科共享统一的数据特征模板，保证了不同配置间的数据格式高度一致，降低了多任务处理的复杂度。
使用方法
研究人员可通过HuggingFace的datasets库直接加载该数据集，利用其多配置特性按需选择特定学科进行模型评估。使用时，需指定配置名称（如'abstract_algebra'）并调用相应split（dev/val/test）。对于俄语或双语模型的评测，可分别提取question_ru或question_en字段作为输入，结合choices_ru或choices_en构造提示模板，并以answer作为监督信号计算准确率。该数据集尤其适用于对比分析模型在英文与俄文语境下的知识迁移能力，或作为多语言预训练模型微调时的标准化验证集。
背景与挑战
背景概述
MMLU（Massive Multitask Language Understanding）数据集由Hendrycks等人于2020年提出，旨在评估语言模型在57个学科领域中的知识广度与推理能力，涵盖人文、社科、理工及医学等多元主题。该数据集因其全面性和挑战性，迅速成为衡量大语言模型综合性能的标杆。NLPCoreTeam/mmlu_ru在此基础上进行了俄语适配，由NLPCore团队创建，将原英文问答与选项精准翻译为俄语，形成双语对照结构。这一工作旨在填补多语言评估的空白，推动俄语自然语言处理的发展，尤其为非英语场景下的模型泛化能力提供了重要测试基准，对跨语言知识迁移研究具有深远影响。
当前挑战
该数据集面临的核心挑战在于确保翻译的语义保真度与文化适配性，避免因语言差异导致的知识失真或歧义。构建过程中，技术难点包括：1）处理俄语复杂的语法变格与词汇形态，使选择题选项保持等效性；2）覆盖57个学科的专业术语翻译，需领域专家介入以维持学术严谨性；3）平衡数据规模与质量，在有限样本（每学科约100-500题）下实现代表性。此外，评估模型时需解决语言偏差问题，即模型可能依赖英语预训练知识而非俄语理解能力，这要求设计严谨的零样本或少样本测试协议，以真实反映跨语言推理水平。
常用场景
经典使用场景
MMLU-RU数据集作为MMLU（Massive Multitask Language Understanding）的俄语扩展版本，经典使用场景在于评估和比较多语言大语言模型在跨语言知识推理任务上的表现。该数据集涵盖了从抽象代数到世界历史等57个学科领域的多项选择题，每个样本均提供英文和俄语双语版本的题目及选项，研究者可借此系统性地衡量模型在俄语语境下对科学、人文、工程等专业知识的掌握程度，尤其适合用于检验预训练语言模型在非英语语言上的泛化能力与知识迁移效果。
解决学术问题
该数据集有效解决了多语言自然语言处理领域中缺乏大规模、高质量、多学科俄语知识评估基准的学术难题。通过提供与英文版MMLU严格对齐的俄语翻译，它使得研究者能够量化分析语言模型在俄语环境中的知识覆盖广度与推理深度，揭示了模型在跨语言知识迁移时可能存在的语义偏差与性能衰减。这一资源为探索语言模型的多语言能力上限、评估不同训练策略对俄语性能的影响，以及推动多语言人工智能系统的公平性研究提供了关键支撑。
衍生相关工作
基于MMLU-RU数据集，研究者已衍生出多项经典工作，包括开发俄语专用的多任务学习模型、设计面向低资源语言的跨语言知识蒸馏方法，以及构建用于分析语言模型在俄语与英语之间知识对齐的对比学习框架。该数据集还催生了针对俄语大模型微调策略的系统性研究，例如探索提示工程与上下文学习在俄语场景下的最优配置。此外，它被广泛用作俄语语言模型排行榜的评测基准，推动了诸如RuGPT、YaLM等俄语预训练模型的迭代与改进。
以上内容由遇见数据集搜集并总结生成
关于我们
遇见数据集——让每个数据集都被发现，让每一次遇见都有价值。
数发科官网www.sfktec.com
联系我们
youjian
selectdataset@iotsh.com.cn
商业合作
数据驱动未来，携手共赢发展
二维码
社区交流群
二维码
科研交流群
商业服务