Supplementary Material for: Development and validation of two bronchoscopy knowledge assessments

Introduction: Simulation-based training (SBT) is a key method for teaching bronchoscopy skills to pulmonology residents. A theoretical foundation can enhance SBT efficiency. This study developed and evaluated the validity of an anatomy and theoretical bronchoscopy exam using Kane’s validity framework.
Methods: 19 anatomy and 58 theoretical exam questions, developed by pulmonology experts, were assessed through two Delphi rounds. Both exams were then taken by 53 prepared pulmonology residents. The theoretical exam was also taken by three unprepared groups: novices, intermediates and experts. Using the residents’ data, scoring evidence for the theoretical exam was evaluated using item difficulty and item discrimination indices, and generalization evidence was assessed using Cronbach’s alpha. Extrapolation evidence was obtained by comparing theoretical exam scores across the different groups. Implications evidence for both exams was gathered by evaluating residents’ preparedness, based on exam performance and instructor feedback.
Results: The Delphi procedure resulted in 19 anatomy and 31 theoretical questions. Item difficulty values predominantly ranged from 0.85-1.0, item discrimination indices mostly ranged from 0.0-0.25. Cronbach’s alpha was 0.55. While scores appeared to correlate with experience, no significant differences were observed between the four groups. Most residents passed both exams on their first attempt, and instructors rated their anatomical knowledge as good.
Conclusion: Expert involvement and acceptable item difficulty, item discrimination and internal consistency supported the exams’ validity. The exams also effectively motivated residents to prepare for SBT. These findings highlight the value of pre-SBT exams in enhancing residents' preparation, allowing more time to focus on mastering procedural skills.