Amharic Facebook Dataset for Hate Speech detection

This dataset is collected from Facebook pages of activists who write their posts using Geez script and comments of their followers. It is collected manually by going through each post and comment. To select activists page and specific posts from those pages we set the following rules 1. Facebook pages should have more than 50,000 like (followers). 2. The post should have more than 300 comments because it shows they have a lot of active participants for their posts. 3. The pages and activists have to use the Amharic language as the main language for their posts. Based on the above rules 30,000 datasets were collected from selected Facebook pages and, all of the unique posts annotated by using ten annotators from different background of culture, religion, and ethnic group, among those ten annotators, five of them given the same number of posts to measure the inter-annotator agreement among them, which achieves Fleiss Kappa of 0.662 and the labeling task is performed based on the guideline given from the researcher. To help the annotator to label posts and comments collected from Facebook guidelines have prepared based on the definition from the UN general definition of hate speech and offensive language and Ethiopian hate speech law by using the annotation style other researchers used. The 30,000 datasets labeled into the binary class of hate and free based on the guideline. Finally, the data cleaning and normalization of Amharic characters are performed, Because In Amharic language characters like ሀ, ኀ, ሐ, ኃ, ኻ, ሓ and ሃ, ሰ and ሠ, ጸ and ፀ, አ, ኣ, ዐ and ዓ represent the same consonants with the same pronunciation and it can be used interchangeably without any meaning differences.

本数据集采集自使用吉兹字母（Geez script）发布帖文的活动家的Facebook主页，以及其粉丝的评论内容。所有数据均通过人工逐条遍历帖文与评论的方式完成采集。为选定符合要求的活动家主页及对应帖文，我们设定了三项筛选规则：其一，目标Facebook主页的点赞数（粉丝量）需超过50000；其二，单条帖文的评论数需超过300，以此体现该帖文拥有大量活跃互动参与者；其三，主页运营者及活动家需以阿姆哈拉语（Amharic）作为帖文的主要发布语言。依据上述规则，我们从筛选出的Facebook主页中共采集得到30000条数据集。所有唯一帖文均由10名来自不同文化、宗教及族裔背景的标注人员完成标注；其中5名标注人员被分配了等量的帖文任务，用于测算标注者间一致性，最终得到的费利斯κ系数（Fleiss Kappa）为0.662。本次标注工作严格遵循研究者预先制定的标注指南开展。为协助标注人员完成帖文与评论的标注，我们结合联合国仇恨言论与冒犯性语言通用定义、埃塞俄比亚仇恨言论相关法律，并参考其他研究者的标注范式，制定了本次标注指南。本次30000条数据集依据上述指南被划分为"仇恨言论"与"非仇恨言论"两个二元类别。最后，我们完成了阿姆哈拉语字符的数据清洗与归一化工作——这是因为阿姆哈拉语中，ሀ、ኀ、ሐ、ኃ、ኻ、ሓ与ሃ，ሰ与ሠ，ጸ与ፀ，አ、ኣ、ዐ与ዓ等字符属于发音一致的辅音，可互换使用且语义无差异。

搜集汇总

背景与挑战

背景概述

该数据集是一个用于仇恨言论检测的阿姆哈拉语Facebook数据集，包含30,000条从埃塞俄比亚活动人士页面收集的帖子和评论，经过多背景标注者标注并达到0.662的Fleiss Kappa一致性。数据经过清洗和字符标准化，适用于自然语言处理任务，遵循CC BY 4.0许可证。