R: a replication script for the analyses.

Background and objectiveMultistate models, which allow the prediction of complex multistate survival processes such as multimorbidity, or recovery, relapse and death following treatment for cancer, are being used for clinical prediction. It is paramount to evaluate the calibration (as well as other metrics) of a risk prediction model before implementation of the model. While there are a number of software applications available for developing multistate models, currently no software exists to aid in assessing the calibration of a multistate model, and as a result evaluation of model performance is uncommon. calibmsm has been developed to fill this gap.MethodsAssessing the calibration of predicted transition probabilities between any two states is made possible through three approaches. The first two utilise calibration techniques for binary and multinomial logistic regression models in combination with inverse probability of censoring weights, whereas the third utilises pseudo-values. All methods are implemented in conjunction with landmarking to allow calibration assessment of predictions made at any time beyond the start of follow up. This study focuses on calibration curves, but the methodological framework also allows estimation of calibration slopes and intercepts.ResultsThis article serves as a guide on how to use calibmsm to assess the calibration of any multistate model, via a comprehensive example evaluating a model developed to predict recovery, adverse events, relapse and survival in patients with blood cancer after a transplantation. The calibration plots indicate that predictions of relapse made at the time of transplant are poorly calibrated, however predictions of death are well calibrated. The calibration of all predictions made at 100 days post transplant appear to be poor, although a larger validation sample is required to make stronger conclusions.Conclusionscalibmsm is an R package which allows users to assess the calibration of predicted transition probabilities from a multistate model. Evaluation of model performance is a key step in the pathway to model implementation, yet evaluation of the performance of predictions from multistate models is not common. We hope availability of software will help model developers evaluate the calibration of models being developed.背景与目标：多状态模型（multistate models）可用于预测复杂的多状态生存过程，例如多病共存，或癌症治疗后的恢复、复发与死亡，目前已被应用于临床预测。在风险预测模型部署前，对其校准度（及其他评估指标）进行评估至关重要。尽管当前已有多款软件可用于开发多状态模型，但尚无工具可辅助评估多状态模型的校准性能，这使得模型性能评估的实践并不常见。calibmsm正是为填补这一空白而开发。 方法：可通过三种方法实现任意两个状态间预测转移概率的校准评估。前两种方法将删失权重逆概率与二项、多项逻辑回归模型的校准技术相结合，第三种方法则利用伪值（pseudo-values）。所有方法均结合地标分析（landmarking），以支持对随访启动后任意时间点生成的预测进行校准评估。本研究聚焦于校准曲线，但该方法框架同样可用于估计校准斜率与截距。 结果：本文为使用calibmsm评估任意多状态模型校准度的实操指南，通过一个全面的案例展开说明——该案例用于评估一款用于预测血液病患者移植后恢复、不良事件、复发与生存的模型。校准图显示，移植时点生成的复发预测校准效果欠佳，但死亡预测的校准表现良好。移植后100天生成的所有预测的校准效果均较差，不过需要更大规模的验证样本才能得出更具说服力的结论。 结论：calibmsm是一款R软件包，可用于评估多状态模型的预测转移概率校准度。模型性能评估是模型部署流程中的关键环节，但目前多状态模型预测性能的评估实践并不普及。我们希望这款软件的推出能够帮助模型开发者对所开发模型的校准度进行系统性评估。