User Study of Explanations for the Outcomes of Goal Recognition Systems

The resource comprises a Qualtrics survey containing the design of a user study aimed at evaluating the effectiveness and acceptability of explanations provided for the outcomes of a Goal Recognition AI system. The study was conducted as part of the PhD thesis Explaining the Outcomes of Goal Recognition Systems, by Jair da Silva Ferreira Junior (2026). In the study, we evaluate participants goal recognition performance and perceptions of Impact and Critique explanations under several experimental settings: Explanatory condition: information presented to participants when assessing the likelihoods of the missions for a particular GR problem. GR prediction correctness: how correct the prediction of the GR is for a particular problem. Scenario group: the six GR problems are split into two scenario groups: A and B. Each group contains three problems, one for each GR prediction correctness category. Presentation order of GR prediction correctness: the order in which the three prediction correctness categories in a scenario group are presented to participants. We divided participants into two cohorts: Between-subjects: participants saw either an Impact or a Critique explanation, then reassessed mission likelihoods, and rated explanatory attributes. Within-subject: participants saw an Impact explanation and reassessed mission likelihoods, followed by a Critique explanation and another reassessment. They then rated explanatory attributes for both explanations, which were presented side by side in randomised left-right order.