Comparison of different Lunit INSIGHT CXR software versions when reading chest radiographs for tuberculosis

New versions of computer-aided detection (CAD) software for chest X-ray
(CXR) interpretation during tuberculosis (TB) screening are regularly
released, which purport to have incremental performance gains. We
independently measured the differences in CAD software performance between
INSIGHT CXR (Lunit, South Korea) versions 3.1.0.0 and 3.9.0.1. A
well-characterized Digital Imaging and Communications in Medicine (DICOM)
test library was compiled using data from a community-based TB screening
initiative in Ho Chi Minh City, Viet Nam. The performance of Lunit CAD
software versions was compared by measuring the area under the receiver
operating characteristic curve (AUC), stratified by key clinical and
demographic variables and using Xpert MTB/RIF Ultra (Ultra) test results
as the reference standard. Median abnormality scores were compared using
the Wilcoxon signed-rank test and performance characteristics were
compared at selected cut-off thresholds between the two versions. The
DICOM test library contained 2,708 participants, of whom 10.3% had a
Mycobacterium tuberculosis (MTB) Positive Ultra test result. The newer
software version had a significantly higher AUC than its predecessor (AUC
0.78 vs 0.76, p=0.029), and performed significantly better among people
with a past history of TB (AUC 0.73 vs 0.67, p=0.003), older individuals
(0.75 vs 0.77, p=0.040) and males (0.73 vs 0.76, p=0.008). The median
abnormality score was significantly higher for the newer software version
(0.61 vs 0.35, p<0.001) and if the newer software’s cut-off
threshold was not optimized, its performance was significantly less
accurate than its predecessors as the same cut-off threshold. Although
INSIGHT CXR v3.9.0.1 has significantly improved performance
characteristics compared to its predecessor, further studies should
prospectively assess how these performance differences translate into
real-world improvements during TB screening. As new CAD software versions
are rolled out, the selection of a cut-off threshold must be assessed and
re-calibrated to ensure the continued accuracy of CXR interpretation.