The development of an instrument assessing the quality of a chatbot

Conversational agents driven by Artificial Intelligence (AI), such as chatbots, are increasingly implemented in behavioral and mental health interventions. However, there is a lack of clear guidance on the design quality and evaluation of the quality of these chatbots. It is therefore timely to develop an instrument that can assess how users perceive the quality of these AI-driven chatbots for behavioral and mental health interventions. In this research project, we will develop a psychometric valid instrument that can assess the quality of AI-driven chatbots for behavioral and mental health interventions. In phase 1, we will formulate defined subscales and items based on literature, existing relevant questionnaires, experience of the research team and prompting AI-driven chatbots. In phase 2, we invite panels of experts and layman to assess these concept subscales, definitions, and items. Based on this feedback, the instrument will be adjusted and, subsequently examined in a pilot study in phase 3. After final adjustments, the structure and psychometric validity of the instrument will be tested in phase 4. This project will result in a psychometric valid instrument that can assess users’ perceived quality of AI-driven chatbots for behavioral and mental health interventions and will be ready to be used in a broad range of research projects.