SaiedAlshahrani/Wikipedia-Corpora-Report

Wikipedia-Corpora-Report数据集是一个元数据数据库，用于支持在线仪表盘，展示人类和机器人如何生成或编辑维基百科版本，并提供所有维基百科版本（320种语言）的页面和编辑指标。这些指标分别统计了文章和非文章的页面数量，以及文章和非文章的编辑次数，按贡献者类型（人类或机器人）分类。元数据从Wikimedia Statistics下载，经过处理后上传到Hugging Face Hub作为数据集。

The Wikipedia-Corpora-Report dataset is a metadata database used to support an online dashboard that illustrates how humans and bots generate or edit Wikipedia editions, providing metrics for pages and edits for all Wikipedia editions (320 languages). These metrics count the number of articles and non-articles, as well as edits on articles and non-articles, categorized by contributor type: humans or bots. The metadata is downloaded from Wikimedia Statistics, processed, and uploaded to the Hugging Face Hub as a dataset.

原始信息汇总

数据集卡片 "Wikipedia-Corpora-Report"

概述

该数据集用作在线WIKIPEDIA CORPORA META REPORT仪表板的元数据数据库，展示人类和机器人如何生成或编辑维基百科版本，并提供所有维基百科版本（320种语言）的“页面”和“编辑”指标。“页面”指标计算文章和非文章，而“编辑”指标统计文章和非文章的编辑，所有这些都按贡献者类型分类：人类或机器人。元数据从Wikimedia Statistics下载，然后处理并上传到Hugging Face Hub作为数据集。

详细信息

更多关于数据集的详细信息，请阅读并引用我们的论文：

bash @inproceedings{alshahrani-etal-2023-performance, title = "{Performance Implications of Using Unrepresentative Corpora in {A}rabic Natural Language Processing}", author = "Alshahrani, Saied and Alshahrani, Norah and Dey, Soumyabrata and Matthews, Jeanna", booktitle = "Proceedings of the The First Arabic Natural Language Processing Conference (ArabicNLP 2023)", month = December, year = "2023", address = "Singapore (Hybrid)", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2023.arabicnlp-1.19", doi = "10.18653/v1/2023.arabicnlp-1.19", pages = "218--231", abstract = "Wikipedia articles are a widely used source of training data for Natural Language Processing (NLP) research, particularly as corpora for low-resource languages like Arabic. However, it is essential to understand the extent to which these corpora reflect the representative contributions of native speakers, especially when many entries in a given language are directly translated from other languages or automatically generated through automated mechanisms. In this paper, we study the performance implications of using inorganic corpora that are not representative of native speakers and are generated through automated techniques such as bot generation or automated template-based translation. The case of the Arabic Wikipedia editions gives a unique case study of this since the Moroccan Arabic Wikipedia edition (ARY) is small but representative, the Egyptian Arabic Wikipedia edition (ARZ) is large but unrepresentative, and the Modern Standard Arabic Wikipedia edition (AR) is both large and more representative. We intrinsically evaluate the performance of two main NLP upstream tasks, namely word representation and language modeling, using word analogy evaluations and fill-mask evaluations using our two newly created datasets: Arab States Analogy Dataset (ASAD) and Masked Arab States Dataset (MASD). We demonstrate that for good NLP performance, we need both large and organic corpora; neither alone is sufficient. We show that producing large corpora through automated means can be a counter-productive, producing models that both perform worse and lack cultural richness and meaningful representation of the Arabic language and its native speakers.", }