prachuryyaIITG/SampurNER

SampurNER是一个细粒度命名实体识别(FgNER)数据集，覆盖了22种印度语言，这些语言被超过20亿人使用。数据集平均包含超过153k个句子、354k个实体和3.3M个标记。该数据集基于实体锚定机器翻译(EaMaTa)框架构建，该框架利用了最大的手动标注英语FgNER数据集FewNERD来创建大规模的多语言FgNER数据集。各种严格的分析和人工评估证实了数据集的高质量，并展示了EaMaTa框架的有效性，与当前最先进技术相比，F1分数提高了9%。

SampurNER is a fine-grained named entity recognition (FgNER) dataset encompassing all 22 scheduled Indian languages spoken by over two billion people across various countries. The dataset, on average, comprises over 153k sentences, 354k entities, and 3.3M tokens in each language. The EaMaTa framework, which leverages the largest manually annotated English FgNER dataset, FewNERD, to create a large-scale FgNER dataset in 22 languages, is used to construct the dataset. Rigorous analyses and human evaluations confirm the high quality of the dataset and demonstrate the effectiveness of the EaMaTa framework with up to 9% increase in F1-score against the current state-of-the-art.