Omartificial-Intelligence-Space/ILMAAM-Arabic-Culturally-Aligned-MMLU

ILMAAM（阿拉伯语言模型多任务评估指数）基准测试为阿拉伯大型语言模型（LLMs）提供了一个文化丰富、语言精炼且上下文相关的评估框架。它基于阿拉伯大规模多任务语言理解（MMLU）数据集，并扩展了与文化对齐的主题和注释，包括流畅性、充分性、文化适宜性、偏见检测、宗教敏感性和社会规范的遵守。该基准测试解决了翻译基准测试中常见的文化和语言挑战，并纳入了对阿拉伯语社区重要的新主题，确保评估与阿拉伯用户的文化规范和期望一致。

The ILMAAM (Index for Language Models for Arabic Assessment on Multitasks) benchmark provides a culturally enriched, linguistically refined, and contextually relevant evaluation framework for Arabic Large Language Models (LLMs). It is based on the Arabic Massive Multitask Language Understanding (MMLU) dataset but extends it with culturally aligned topics and annotations for fluency, adequacy, cultural appropriateness, bias detection, religious sensitivity, and adherence to social norms. Additionally, the benchmark adds five new topics to reflect the unique cultural, historical, and ethical values of Arabic-speaking communities. The annotation process involved a team of eleven experts who reviewed over 2,500 questions to ensure cultural appropriateness, fluency, adequacy, and alignment with Arabic social norms.