Using standardised patients to measure physicians' practice: validation study using audio recordings

OBJECTIVE: To assess the validity of standardised patients to measure the quality of physicians' practice. DESIGN: Validation study of standardised patients' assessments. Physicians saw unannounced standardised patients presenting with common outpatient conditions. The standardised patients covertly tape recorded their visit and completed a checklist of quality criteria immediately afterwards. Their assessments were compared against independent assessments of the recordings by a trained medical records abstractor. SETTING: Four general internal medicine primary care clinics in California. PARTICIPANTS: 144 randomly selected consenting physicians. MAIN OUTCOME MEASURES: Rates of agreement between the patients' assessments and independent assessment. RESULTS: 40 visits, one per standardised patient, were recorded. The overall rate of agreement between the standardised patients' checklists and the independent assessment of the audio transcripts was 91% (κ=0.81). Disaggregating the data by medical condition, site, level of physicians' training, and domain (stage of the consultation) gave similar rates of agreement. Sensitivity of the standardised patients' assessments was 95%, and specificity was 85%. The area under the receiver operator characteristic curve was 90%. CONCLUSIONS: Standardised patients' assessments seem to be a valid measure of the quality of physicians' care for a variety of common medical conditions in actual outpatient settings. Properly trained standardised patients compare well with independent assessment of recordings of the consultations and may justify their use as a “gold standard” in comparing the quality of care across sites or evaluating data obtained from other sources, such as medical records and clinical vignettes.

研究目的：评估标准化患者（standardised patients）用于衡量医师执业质量的有效性。
研究设计：标准化患者评估验证性研究。医师接诊未提前告知的标准化患者，后者会表现出常见门诊病症。标准化患者会秘密录制就诊全过程，并在就诊结束后立即填写质量标准核查表。随后将其评估结果与经过培训的医疗记录摘要员对就诊录音的独立评估结果进行比对。
研究场景：加利福尼亚州4家普通内科初级保健诊所。
研究对象：144名经随机选取并签署知情同意书的医师。
主要结局指标：患者评估与独立评估之间的一致性率。
研究结果：共完成40次就诊记录，每位标准化患者对应1次就诊。标准化患者填写的核查表与就诊音频转录本的独立评估结果总体一致性率为91%（κ=0.81）。按病症类型、就诊地点、医师培训层级以及咨询环节对数据进行细分后，所得一致性率与总体结果相近。标准化患者评估的灵敏度为95%，特异度为85%。受试者工作特征（Receiver Operator Characteristic，ROC）曲线下面积为90%。
研究结论：标准化患者评估可有效衡量实际门诊场景下多种常见病症的医师诊疗质量。经过规范培训的标准化患者的评估结果与就诊录音的独立评估结果一致性良好，可将其作为“金标准”，用于跨地点诊疗质量对比，或评估医疗记录、临床情景案例（clinical vignettes）等其他来源获取的数据。