Fleiss kappa for doc-2-doc relevance assessment

Here we present a table summarizing the Fleiss’ kappa results. The Fleiss’ Kappa was calculated as a way to measure the degree of agreement between four annotators who evaluated the relevance of a set of documents (15 evaluation articles) regarding its corresponding “reference article”. The table contains 7 columns, the first one presents the topics, 8 in total. The second column shows the “reference articles”, represented by their PubMed-ID and organized by topic. The third column shows the Fleiss’ Kappa results. The fourth column shows the interpretation of the Fleiss' Kappa results being: i) “Poor” results &lt;0.20, ii) “Fair” results within 0.21 - 0.40, and iii) “Moderate” results within 0.41 - 0.60. The fifth column shows the PubMed-IDs of evaluation articles rated by the four annotators as “Relevant” regarding its corresponding “reference article”. The sixth column shows the PubMed-IDs of evaluation articles rated by the four annotators as “Partially relevant” regarding its corresponding “reference article”. The seventh column shows the PubMed-IDs of evaluation articles rated by the four annotators as “Non-relevant” regarding its corresponding “reference article”. <strong>Acknowledgements</strong> <br> This work is part of the STELLA project funded by DFG (project no. 407518790). This work was supported by the BMBF-funded de.NBI Cloud within the German Network for Bioinformatics Infrastructure (de.NBI) (031A532B, 031A533A, 031A533B, 031A534A, 031A535A, 031A537A, 031A537B, 031A537C, 031A537D, 031A538A).<br>本研究呈现一张汇总弗莱伊斯kappa系数（Fleiss’ Kappa）结果的表格。本研究通过计算弗莱伊斯kappa系数，以衡量四位标注者针对15篇评估文献与其对应“参考文献”的相关性评估结果的一致程度。该表格共设7列：第一列为主题，总计8个主题；第二列为以PubMed-ID标识的参考文献，并按主题分类；第三列为弗莱伊斯kappa系数的计算结果；第四列为弗莱伊斯kappa系数结果的解读标准：i） 较差（Poor）：系数值<0.20；ii） 一般（Fair）：系数值介于0.21至0.40之间；iii） 中等（Moderate）：系数值介于0.41至0.60之间；第五列为针对对应“参考文献”被四位标注者判定为“相关”的评估文献的PubMed-ID；第六列为针对对应“参考文献”被四位标注者判定为“部分相关”的评估文献的PubMed-ID；第七列为针对对应“参考文献”被四位标注者判定为“不相关”的评估文献的PubMed-ID。<strong>致谢</strong><br>本研究属于由德国研究基金会（DFG，Deutsche Forschungsgemeinschaft）资助的STELLA项目，项目编号为407518790。本研究得到了德国生物信息基础设施网络（de.NBI，German Network for Bioinformatics Infrastructure）内由德国联邦教育与研究部（BMBF，Bundesministerium für Bildung und Forschung）资助的de.NBI云项目支持，资助编号包括031A532B、031A533A、031A533B、031A534A、031A535A、031A537A、031A537B、031A537C、031A537D及031A538A。