Key Metrics and Definitions.

AI has been proposed as a triage or “rule-out” device to reduce radiologist workload, but it is presently unclear how an AI “rule-out” threshold should be determined. We present a framework for determining an optimal threshold. Using a retrospective study design, 114,229 bilateral 2D digital screening mammograms were analyzed from 2006-2023 at a single study site. All mammograms were given an AI score using Mirai, an open-source deep-learning model which provides a 1-year risk score. Several metrics were examined using two thresholds for determining ruled out versus retained cases: 1) Caseload Reduction Rate (CRR; percent of caseload reduced due to rule-out), 2) Gross AI False Omission Rate (G-FOR; probability of a patient having breast cancer if ruled out), 3) AI Net False Omission Rate (N-FOR; probability of a patient having breast cancer if ruled out and the radiologist would have caught in standard care [i.e., no triage]), 4) AI Adjusted Net False Omission Rate (30%) (AN-FOR[30%]; N-FOR adjusted for the hypothetical scenario where radiologists detect an extra 30% of breast cancers among AI retained cases). The two thresholds were risk scores of 0.2 (Youden’s J) and 0.05 (AN-FOR[30%]=0). The former is mathematically optimal; the latter reflects a threshold where AI “rule-out” does not introduce any total increase in False Negatives. At the 0.20 threshold, G-FOR, N-FOR, and AN-FOR (30%) are 0.26%, 0.17%, and 0.14%, respectively (223, 141, and 121, respectively, missed cancer cases) and CRR = 75%. At the 0.05 threshold, the G-FOR, N-FOR, and AN-FOR (30%) are 0.12%, 0.07%, and 0.00% (49, 30, and 0, respectively, missed cancer cases) and CRR = 36%. We demonstrate how radiology practices can consider the trade-offs of using different AI scores as “rule-out” thresholds. At the AN-FOR rate of 30%, the Youden’s J threshold results in 121 additional missed cancers for a 75% caseload reduction. We estimate no additional missed cancers at a 36% caseload reduction.

人工智能（AI）已被提议作为分诊或‘排除’设备，以减轻放射科医师的工作负担，但目前尚不清楚应如何确定AI‘排除’阈值。我们提出了一种确定最优阈值的研究框架。本研究采用回顾性研究设计，对2006年至2023年间某单一研究中心的114229份双侧二维数字化筛查乳腺X线摄影影像进行了分析。所有乳腺X线摄影影像均通过开源深度学习模型Mirai生成AI评分，该模型可提供1年发病风险评分。本研究通过两个阈值判定排除病例与留存病例，并对多项指标进行了评估：1） 病例量缩减率（Caseload Reduction Rate, CRR）：因AI排除操作而缩减的病例占比；2） AI总假漏诊率（Gross AI False Omission Rate, G-FOR）：被AI判定为排除的患者罹患乳腺癌的概率；3） AI净假漏诊率（AI Net False Omission Rate, N-FOR）：被AI判定为排除，且在标准诊疗流程（即无分诊模式）下本可被放射科医师检出的患者罹患乳腺癌的概率；4） AI校正净假漏诊率（30%）（AI Adjusted Net False Omission Rate [30%], AN-FOR[30%]）：针对放射科医师在AI留存病例中额外检出30%乳腺癌的假设场景，对N-FOR进行校正后得到的指标。两个阈值分别为风险评分0.2（尤登J指数，Youden’s J）与0.05（AN-FOR[30%]=0）。前者在数学层面为最优阈值；后者对应的阈值下，AI‘排除’操作不会导致假阴性总数出现额外增加。当阈值为0.20时，G-FOR、N-FOR与AN-FOR（30%）分别为0.26%、0.17%与0.14%（对应漏诊癌症病例数分别为223、141与121），病例量缩减率为75%。当阈值为0.05时，G-FOR、N-FOR与AN-FOR（30%）分别为0.12%、0.07%与0.00%（对应漏诊癌症病例数分别为49、30与0），病例量缩减率为36%。本研究展示了放射科实践中如何权衡采用不同AI评分作为‘排除’阈值的利弊得失。在AN-FOR（30%）场景下，采用尤登J指数阈值可实现75%的病例量缩减，但会额外漏诊121例癌症病例。我们估算，当病例量缩减率为36%时，不会出现额外的癌症漏诊情况。