OntoValidate: OntoNotes 5.0 NER validation dataset

OntoValidate dataset consists of 603 randomly chosen raw textsfrom the original OntoNote 5.0 dataset (3637 raw texts in total).
All raw texts are split into 13 subsets. Each subset was manuallyannotated with named entities by at least 10 people. Each annotator followed the official annotation guidelines (OntoNotes Named Entity Guidelines VERSION 14.0)that are distributed together with the OntoNotes 5.0 dataset. 
The OntoValidate dataset allows for a comparison of how different peopleunderstand the named entity recognition task. On the other hand, it also gave ussome insights about the precision of annotation guidelines itself.
Format
Each directory of the OntoValidate dataset represents a separate annotator.The first number in the directory name is the sequence number of the subset, the second isthe sequence number of the annotator. Each directory contains the JSON files that containthe annotations in spaCy JSON format. 
All annotations were created using the NER Text Annotator tool (https://tecoholic.github.io/ner-annotator/).

OntoValidate数据集由从原始OntoNote5.0数据集（共3637条原始文本）中随机选取的603条原始文本构成。所有原始文本被划分为13个子集，每个子集至少由10名标注人员手动进行命名实体标注。每位标注者均遵循随OntoNotes5.0数据集一同分发的官方标注指南（OntoNotes Named Entity Guidelines VERSION14.0）。

Format
OntoValidate数据集的每个目录代表一位独立标注者。目录名称中的第一个数字为子集序号，第二个数字为标注者序号。每个目录包含采用spaCy JSON格式的标注文件。
所有标注均通过NER Text Annotator工具（https://tecoholic.github.io/ner-annotator/）创建。

OntoValidate数据集可用于比较不同人员对命名实体识别（Named Entity Recognition）任务的理解差异；另一方面，它也为我们提供了关于标注指南本身精确性的若干见解。