OntoValidate: OntoNotes 5.0 NER validation dataset

OntoValidate dataset consists of 603 randomly chosen raw textsfrom the original OntoNote 5.0 dataset (3637 raw texts in total).
All raw texts are split into 13 subsets. Each subset was manuallyannotated with named entities by at least 10 people. Each annotator followed the official annotation guidelines (OntoNotes Named Entity Guidelines VERSION 14.0)that are distributed together with the OntoNotes 5.0 dataset. 
The OntoValidate dataset allows for a comparison of how different peopleunderstand the named entity recognition task. On the other hand, it also gave ussome insights about the precision of annotation guidelines itself.
Format
Each directory of the OntoValidate dataset represents a separate annotator.The first number in the directory name is the sequence number of the subset, the second isthe sequence number of the annotator. Each directory contains the JSON files that containthe annotations in spaCy JSON format. 
All annotations were created using the NER Text Annotator tool (https://tecoholic.github.io/ner-annotator/).

OntoValidate数据集由从原始OntoNote 5.0数据集（共3637条原始文本）中随机选取的603条原始文本组成。
所有原始文本被划分为13个子集。每个子集由至少10人手动标注命名实体。每位标注员均遵循与OntoNotes 5.0数据集一同分发的官方标注指南（OntoNotes Named Entity Guidelines VERSION14.0）。
OntoValidate数据集可用于比较不同人群对命名实体识别任务（Named Entity Recognition Task）的理解差异；另一方面，它也为我们提供了关于标注指南本身精确性的洞见。
格式
每个OntoValidate数据集目录代表一个独立标注员。目录名称中的第一个数字是子集序号，第二个数字是标注员序号。每个目录包含采用spaCy JSON格式的标注文件。
所有标注均使用NER文本标注工具（Named Entity Recognition Text Annotator，https://tecoholic.github.io/ner-annotator/）创建。