Data_Sheet_5_Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc.CSV

Quality control is a critical step in the processing and analysis of functional magnetic resonance imaging data. Its purpose is to remove problematic data that could otherwise lead to downstream errors in the analysis and reporting of results. The manual inspection of data can be a laborious and error-prone process that is susceptible to human error. The development of automated tools aims to mitigate these issues. One such tool is pyfMRIqc, which we previously developed as a user-friendly method for assessing data quality. Yet, these methods still generate output that requires subjective interpretations about whether the quality of a given dataset meets an acceptable standard for further analysis. Here we present a quality control protocol using pyfMRIqc and assess the inter-rater reliability of four independent raters using this protocol for data from the fMRI Open QC project (https://osf.io/qaesm/). Data were classified by raters as either “include,” “uncertain,” or “exclude.” There was moderate to substantial agreement between raters for “include” and “exclude,” but little to no agreement for “uncertain.” In most cases only a single rater used the “uncertain” classification for a given participant’s data, with the remaining raters showing agreement for “include”/“exclude” decisions in all but one case. We suggest several approaches to increase rater agreement and reduce disagreement for “uncertain” cases, aiding classification consistency.

功能磁共振成像（functional magnetic resonance imaging，fMRI）数据的处理与分析流程中，质量控制是至关重要的一环。其核心目标是剔除存在质量问题的数据，避免此类数据在后续分析与结果报告中引发误差。人工审核数据的过程往往耗时费力且极易出错，受人为失误影响较大。自动化质量控制工具的研发正是为了缓解上述问题。其中一款工具为pyfMRIqc——我们此前开发的一款便于用户使用的数据质量评估方法。但此类工具生成的评估结果仍需研究人员进行主观判断，以确定特定数据集的质量是否达到可用于后续分析的合格标准。本研究提出了一套基于pyfMRIqc的质量控制流程，并借助该流程，针对来自fMRI开放质量控制项目（https://osf.io/qaesm/）的数据集，评估了4名独立评分者的评分者间信度。评分者将数据集划分为“纳入”“不确定”与“排除”三类。评分者在“纳入”与“排除”两类的判定上达成了中等至极强的一致性，但在“不确定”类的判定上几乎没有一致性。多数情况下，仅1名评分者会将某一被试的数据判定为“不确定”，其余评分者均会就“纳入”/“排除”的判定达成一致，仅存在1例例外。本研究提出了若干方案，以提升评分者间的一致性，减少“不确定”类判定的分歧，从而提升分类结果的一致性。