tiiuae/alyah-emirati-benchmark

Alyah（在Emirati语中意为北极星）是一个手动策划的评估基准，用于评估阿拉伯语大型语言模型在Emirati方言上的表现。它旨在不仅衡量语言能力，还包括文化意识、实用理解和对Emirati阿拉伯语日常表达中社会规范的敏感性。该基准包含1,173个多项选择题，每个问题有四个候选答案和一个正确答案。Alyah针对需要方言流畅性和基于文化的推理的场景，超越了现代标准阿拉伯语（MSA）。

Alyah (الياه in Emirati means North Star) is a manually curated evaluation benchmark for assessing the performance of Arabic large language models on the Emirati dialect. It is designed to measure not only linguistic competence, but also cultural awareness, pragmatic understanding, and sensitivity to social norms as expressed in everyday Emirati Arabic. The benchmark consists of 1,173 multiple-choice questions, each with four candidate answers and exactly one correct option. Alyah targets scenarios where dialectal fluency and culturally grounded reasoning are required, going beyond Modern Standard Arabic (MSA).