Supplementary material

Contextualization of questions has been suggested to deal with the capabilities of generative AI in an online environment. For this paper we experimented with contextualization to observe how generative AI (in this instance ChatGPT) deals with different elements of contextualization. The experiment used a hypothetical case study for a tailored course in Enterprise Architecture at postgraduate level and looked at contextualization along two axes: (1) Contextualized questioning, and (2) providing different learner contexts in the form of personas to ChatGPT and seeing whether ChatGPT adapted its responses. The contextualized questioning aspects used three different assessments. For the first assessment, individual questions on Enterprise Architecture were asked. For the second and third assessments the case study was used. In the second assessment a general essay that addresses certain elements of Enterprise Architecture was requested, while in the third assessment structured questions on the case study were asked. Responses for each of the assessments were independently marked by the authors, with marking results compared and discussed. The experiment found differences in the quality of responses generated by ChatGPT related to contextualization and degree of structuring in the questions, and in the responses provided to the different personas. Differences in responses to the personas were however not contextualized. It was observed that ChatGPT produced higher quality answers to individually structured questions but fared poorly when prompted for an essay. Educators should therefore contextualize and structure questions in the form of case studies which stimulate learner creativity and critical thinking skills.