The annotator agreement and the overall model performance.

Two measures are used: ordinal Krippendorff’s Alpha and accuracy (Acc). The first line is the self-agreement of individual annotators, and the second line is the inter-annotator agreement between different annotators. The last two lines are the model evaluation results, on the training and the out-of-sample evaluation sets, respectively. Note that the overall model performance is comparable to the inter-annotator agreement.

本研究采用两项评价指标：有序克里彭多夫阿尔法系数（ordinal Krippendorff’s Alpha）与准确率（Accuracy，Acc）。首行代表单个标注者的自一致性，次行代表不同标注者间的标注者间一致性。最后两行分别为模型在训练集与样本外评估集上的评估结果。需注意，模型的整体性能与标注者间一致性水平相当。