Clinical Validity of Caries Vignettes Requires More Than Reliability

Objectives To determine, in the same dentists, whether standardized caries cases reliably distinguish treatment tendencies, whether those tendencies recur in practice, and whether vignette-derived probabilities reproduce clinical treatment rates. Methods This same-dentist secondary analysis linked vignette responses, including a common 16-case factorial set, with pre-intervention and intervention clinical records. We estimated the reliability of each dentist’s overall vignette treatment tendency, tested whether that tendency recurred in planned treatment and recorded lesion opening, and evaluated whether vignette probabilities reproduced clinical treatment rates in dentists excluded from model fitting. Interpretable models examined context-related changes in cue–decision relationships, with familywise adjustment across eight cue-phase comparisons. Results Overall vignette treatment tendency was reliable (median, 0.955), whereas cue-specific reliabilities ranged from 0.260 to 0.523. It was moderately correlated with clinical treatment rates before (Spearman rho, 0.418; 95% CI, 0.233 to 0.583) and during the intervention (rho, 0.383; 95% CI, 0.184 to 0.550). Across a 2-SD tendency contrast, adjusted opening rates differed by approximately 22 to 28 per 100 lesions across the two clinical phases. Direct vignette probabilities were miscalibrated (Brier score, 0.26762); recalibration improved the score to 0.22823 and closed 59.3% of the observed prediction-error gap. Models trained on clinical records reached Brier scores of 0.200 to 0.201, and adding vignette records produced no demonstrated improvement. The shiny-versus-chalky treatment contrast was 19 to 21 percentage points smaller in clinical records than in vignettes and was the only cue difference supported after familywise adjustment. Conclusions Validity was use-specific: standardized caries cases reliably distinguished dentists, and the measured tendency was reflected in clinical decisions, but vignette probabilities did not reproduce clinical treatment rates. Practitioner comparison and estimation of absolute treatment rates therefore require different validation evidence.