Grading of recommendations assessment, development and evaluation working group data, 2016-2020

This data comes from a study to explore expert consensus on the importance of criteria for assessing certainty of evidence in systematic reviews on the effects of complex interventions. We conducted an online modified-Delphi process from March to May 2017 with 116 international experts in complex intervention research. Participants reviewed, rated, and discussed the importance of 50 criteria derived from a review of systems for assessing certainty of evidence on intervention effects. We analysed quantitative rating data using the RAND/UCLA Appropriateness Method and qualitative data using thematic analysis to identify areas of agreement on the importance of criteria. There was no significant disagreement among participants on the importance of any criterion. Sixteen criteria (32%) were determined to be critically important and had an interquartile range above the “critically important” threshold. Indirectness of the Evidence Base was the only domain for which all criteria were rated as critically important, while Initial Certainty Rating: Study Design was the only domain containing a criterion determined to be of limited importance. The remaining domains had a mix of criteria rated as either critically important or important but not critical. Participants’ comments produced several themes on intervention complexity and certainty criteria that elucidate the rationale for their ratings. Findings provide insight into experts views on how to incorporate a complexity perspective when assessing certainty of evidence in systematic reviews estimating the effects of interventions in public health. Findings from this study will inform ongoing initiatives for developing GRADE guidance specifically for complex interventions.