Data supplementing the MSc thesis: Supporting the expert judgement process for quantifying uncertainties in policy research – A case study of the unique challenges with implementing structured expert elicitation for projecting policy effects

Expert opinions are crucial to quantify unknown variables when it is not feasible to use traditional research methods (limited resources or questions about future values). Implementing structured expert elicitation (SEE) is important for the quality and credibility of assumptions made by these experts or the researchers who enlist them. However, application in practice has been limited.<br>When research is performed to project policy effects, the context is so volatile that the accuracy of experts' judgements cannot be validated. Moreover, experts may focus on quantifying uncertainty while ignoring extreme uncertainties (war, future technological inventions), operating from a status quo point of view, in order to support the policy relevance of their research. This leads to a unique context of the expert judgement process. It is unclear how experts address expert judgement in this practice. Moreover, it is not yet known what problems may limit application of SEE protocols in this context.<br>I work with the PBL Netherlands Environmental Assessment Agency to evaluate their expert judgement process for the KEV (The Climate and Energy Outlook). On a yearly basis their (sector) experts are required to judge dozens of uncertain variables for the projections of the progress on climate and energy goals. This includes policy effects, external factors like weather, and also scientific uncertainties like the inherent uncertainty of the models that are used for the projections.<br><em>How can practitioners in the policy research field be supported in implementing structured expert elicitation to improve their expert judgement?</em><br>To answer this research question and address it through communication design, I followed the Double Diamond method for design-based research (Discover-Define-Develop-Deliver). For the duration of a year, I was an intern at PBL, allowing me to observe and discuss their methodology, while inquiring after what problems they face. I helped organise a pilot for implementation of the IDEA protocol for structured expert elicitation. To evaluate this pilot I designed a focus group session with all different actors involved in the KEV project. Next, to discuss challenges and develop possible solutions I organised a co-design workshop with (sector) experts from different teams that work on the KEV.<br>The results show that experts face several unique challenges in their expert judgement process, which are not directly addressed by SEE protocols. The IDEA protocol shows promise to improve quality of judgements and transparency through increased documentation. However, the protocol lacks support experts say they require to select what uncertainties to address in their research, as well as adequate support to quantify their uncertainty interval.<br>Siloed teams lead to diverging methodology and definitions, as well as missing overall problem demarcation. Because there is limited opportunity for feedback on the accuracy of experts' judgements, there is no universal measure of what makes a judgement 'good'. <br>To address the problems faced by the practitioners at PBL, I designed an instructional video. It informs experts about SEE and provides tangible examples and actionable steps the practitioners can take on the short and long term, on the individual and organisational level. This provides the researchers with the knowledge and control to improve their own practice. At the same time, the video also acknowledges the unique challenges the experts face and reassures them of the confidence they can have in their practice to make decisions. By targeting both teamleaders and sector experts, the facilitating party and the practicing party are involved and given agency in this process.<br>Therefore, research requiring expert judgement on policy uncertainties presents some unique challenges in the expert judgement process. When accuracy cannot be used as a measure of quality, it is important that the researchers invest in the credibility of the methodology. SEE can provide structure and transparency, but additional instructions and methods are needed to support experts to confidently make judgements that balance accuracy and information.<br>Though this case study focuses on a specific organisation and group of experts, the lessons learned from this study indicate a variety of challenges when applying SEE in different research domains and contexts. This could possible indicate the stagnated implementation in practice. Additionally, this research and the development of structured expert elicitation for policy assessment support credibility of expert judgement in this political environment.