SaiedAlshahrani/MASD

---
license: mit
language:
- ar
pretty_name: MASD
size_categories:
- n<1K
---

# Dataset Card for "Masked Arab States Dataset (MASD)"
This dataset is created using 20 Arab States¹ with their corresponding capital cities, nationalities, currencies, and on which continents they are located, consisting of four categories: country-capital
prompts, country-currency prompts, country-nationality prompts, and country-continent prompts. Each prompts category has 40 masked prompts, and the total number of masked prompts in the MASD dataset is 160. This dataset is used to evaluate these Arabic Masked Language Models (MLMs):
1. [SaiedAlshahrani/arwiki_20230101_roberta_mlm_bots](https://huggingface.co/SaiedAlshahrani/arwiki_20230101_roberta_mlm_bots).
2. [SaiedAlshahrani/arwiki_20230101_roberta_mlm_nobots](https://huggingface.co/SaiedAlshahrani/arwiki_20230101_roberta_mlm_nobots).
3. [SaiedAlshahrani/arzwiki_20230101_roberta_mlm](https://huggingface.co/SaiedAlshahrani/arzwiki_20230101_roberta_mlm).
4. [SaiedAlshahrani/arywiki_20230101_roberta_mlm_bots](https://huggingface.co/SaiedAlshahrani/arywiki_20230101_roberta_mlm_bots).
5. [SaiedAlshahrani/arywiki_20230101_roberta_mlm_nobots](https://huggingface.co/SaiedAlshahrani/arywiki_20230101_roberta_mlm_nobots).

For more details about the dataset, please **read** and **cite** our paper:

```bash
@inproceedings{alshahrani-etal-2023-performance,
title = "{Performance Implications of Using Unrepresentative Corpora in {A}rabic Natural Language Processing}",
author = "Alshahrani, Saied and Alshahrani, Norah and Dey, Soumyabrata and Matthews, Jeanna",
booktitle = "Proceedings of the The First Arabic Natural Language Processing Conference (ArabicNLP 2023)",
month = December,
year = "2023",
address = "Singapore (Hybrid)",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.arabicnlp-1.19",
doi = "10.18653/v1/2023.arabicnlp-1.19",
pages = "218--231",
abstract = "Wikipedia articles are a widely used source of training data for Natural Language Processing (NLP) research, particularly as corpora for low-resource languages like Arabic. However, it is essential to understand the extent to which these corpora reflect the representative contributions of native speakers, especially when many entries in a given language are directly translated from other languages or automatically generated through automated mechanisms. In this paper, we study the performance implications of using inorganic corpora that are not representative of native speakers and are generated through automated techniques such as bot generation or automated template-based translation. The case of the Arabic Wikipedia editions gives a unique case study of this since the Moroccan Arabic Wikipedia edition (ARY) is small but representative, the Egyptian Arabic Wikipedia edition (ARZ) is large but unrepresentative, and the Modern Standard Arabic Wikipedia edition (AR) is both large and more representative. We intrinsically evaluate the performance of two main NLP upstream tasks, namely word representation and language modeling, using word analogy evaluations and fill-mask evaluations using our two newly created datasets: Arab States Analogy Dataset (ASAD) and Masked Arab States Dataset (MASD). We demonstrate that for good NLP performance, we need both large and organic corpora; neither alone is sufficient. We show that producing large corpora through automated means can be a counter-productive, producing models that both perform worse and lack cultural richness and meaningful representation of the Arabic language and its native speakers.",
}
```

_{1. We only drop two Arab states: the United Arab Emirates (الإمارات العربية المتحدة) and Comoros (جزر القمر), because they or their capital cities are written as open compound words (two words), which cannot be directly handled by the word embedding models, like Abu Dhabi (أبو ظبي).}

原始信息汇总

数据集卡片 for "Masked Arab States Dataset (MASD)"

数据集概述

• 名称: Masked Arab States Dataset (MASD)
• 语言: 阿拉伯语
• 许可证: MIT
• 数据集大小: n<1K
• 数据集描述: 该数据集包含20个阿拉伯国家的首都、国籍、货币和所在大陆的信息，分为四个类别：国家-首都提示、国家-货币提示、国家-国籍提示和国家-大陆提示。每个类别包含40个掩码提示，总共有160个掩码提示。

数据集用途

• 评估对象: 用于评估以下阿拉伯掩码语言模型（MLMs）：
  1. SaiedAlshahrani/arwiki_20230101_roberta_mlm_bots
  2. SaiedAlshahrani/arwiki_20230101_roberta_mlm_nobots
  3. SaiedAlshahrani/arzwiki_20230101_roberta_mlm
  4. SaiedAlshahrani/arywiki_20230101_roberta_mlm_bots
  5. SaiedAlshahrani/arywiki_20230101_roberta_mlm_nobots

参考文献

• 论文: Performance Implications of Using Unrepresentative Corpora in Arabic Natural Language Processing
• 作者: Alshahrani, Saied; Alshahrani, Norah; Dey, Soumyabrata; Matthews, Jeanna
• 会议: Proceedings of the The First Arabic Natural Language Processing Conference (ArabicNLP 2023)
• 出版商: Association for Computational Linguistics
• 页码: 218--231

数据集细节

• 排除国家: 排除了两个阿拉伯国家：阿拉伯联合酋长国（الإمارات العربية المتحدة）和科摩罗（جزر القمر），因为它们或其首都城市是以开放复合词（两个词）的形式书写，无法直接由词嵌入模型处理，例如阿布扎比（أبو ظبي）。