DiaLex

DiaLex是由不列颠哥伦比亚大学自然语言处理实验室创建的一个用于评估多语种阿拉伯语词向量的基准数据集。该数据集涵盖了五个主要的阿拉伯语方言，包括阿尔及利亚、埃及、黎巴嫩、叙利亚和突尼斯方言，共包含3070对单词，均匀分布在每个方言中。DiaLex用于测试六种语法和语义关系，如男性对女性、单数对复数等。数据集的创建过程涉及由至少一名母语者手动生成每个方言的单词对。DiaLex不仅用于评估词向量，还支持将方言整合到阿拉伯语课程中，解决方言处理中的评估问题。DiaLex is a benchmark dataset developed by the Natural Language Processing Lab at the University of British Columbia for evaluating word embeddings of multi-dialectal Arabic. This dataset covers five major Arabic dialects: Algerian, Egyptian, Lebanese, Syrian, and Tunisian, and comprises 3070 word pairs evenly distributed across each dialect. DiaLex is designed to test six grammatical and semantic relations, including masculine-feminine, singular-plural, and other analogous pairs. The dataset creation process involved manually generating word pairs for each dialect by at least one native speaker. Beyond evaluating word embeddings, DiaLex also supports the integration of Arabic dialects into Arabic language curricula, addressing evaluation challenges in dialectal natural language processing.