Data and analysis underlying the research into the Artificial-Social-Agent Questionnaire: Establishing the long and short questionnaire versions

We present a technical report, data and analysis underlying the study on establishing the Artificial Social Agent (ASA) Questionnaire. The ASA Questionnaire is an instrument for evaluating human interaction with an ASA, resulted from multi-year efforts involving more than 100 Intelligent Virtual Agent (IVA) researchers worldwide. It has 19 measurement constructs constituted by 90 items, which capture more than 80% of the constructs identified in empirical studies published in the IVA conference 2013-2018. This paper reports on construct validity analysis, specifically convergent and discriminant validity of initial 131 instrument items that involved 532 crowd-workers who were asked to rate human interaction with 14 different ASAs. The analysis included several factor analysis models, and resulted in the selection of 90 items for inclusion of the long version of the ASA questionnaire. In addition, a representative item of each construct or dimension was select to create a 24-item short version of the ASA questionnaire. Whereas the long version is suitable for a comprehensive evaluation of human-ASA interaction, the short version allows quick analysis and description of the interaction with the ASA. To support reporting ASA questionnaire results, we also put forward an ASA chart. The chart provides a quick overview of agent profile.

本文呈现了一项关于构建人工社会智能体（Artificial Social Agent，ASA）问卷的研究背后的技术报告、数据及分析。ASA问卷是评估人类与ASA交互的工具，由全球100余名智能虚拟智能体（Intelligent Virtual Agent，IVA）研究者历经多年合作开发而成。该问卷包含19个测量维度，由90个条目构成，覆盖了2013-2018年IVA会议发表的实证研究中所识别维度的80%以上。本文报告了结构效度分析结果，具体针对初始131个条目（由532名众包工作者参与评估，他们需对人类与14种不同ASA的交互进行评分）的聚合效度与区分效度。分析过程采用多种因子分析模型，最终筛选出90个条目组成ASA问卷的长版。此外，通过选取每个维度的代表性条目，构建了包含24个条目的ASA问卷短版。长版适用于人类与ASA交互的全面评估，而短版则可实现对该交互的快速分析与描述。为便于呈现ASA问卷的结果，我们还提出了ASA图谱，该图谱可快速展示智能体的概况。