A Machine Learning Framework for Detecting and Adjusting for Learning Effects in the Evaluation of Medical Device Safety

Medical device failures have resulted in significant patient injury and represent an important preventable health risk in the United States. For example, over the past 10 years the replacement of seven types of faulty cardiovascular devices has cost the healthcare system ~$1.5 billion and impacted thousands of lives. Currently, assurance of post-approval device safety relies extensively on spontaneous reporting of adverse events by providers and device manufacturers; however, it is estimated that <0.5% of device failures are actually reported. In response, the United States Food and Drug Administration has signaled an interest in prospective active surveillance methodologies, which address several of the limitations of passive adverse event reporting and support direct estimation of event rates. Interpreting comparative safety and effectiveness of medical devices using real-world data is challenging and has important methodological gaps. Various patient and provider-level factors might affect adverse events rates of a device implantation procedure. For example, improvements in physician technical performance as a result of device familiarity, a “learning curve”, may influence adverse event rates following device implantation. Routine collection of clinical data during device placement and over the course of patient follow-up may support disentangling learning effects from intrinsic risk of medical devices.  Using observational data poses challenges because of complex patient characteristics resulting in non-linear relationships between risk factors and outcomes. In this work, we sought to develop a machine learning framework to separate learning effects from device signals that can be used in downstream surveillance activities.