Data and analysis underlying the research into the Artificial-Social-Agent Questionnaire: Establishing the long and short questionnaire versions

We present a technical report, data and analysis underlying the study on establishing the Artificial Social Agent (ASA) Questionnaire. The ASA Questionnaire is an instrument for evaluating human interaction with an ASA, resulted from multi-year efforts involving more than 100 Intelligent Virtual Agent (IVA) researchers worldwide. It has 19 measurement constructs constituted by 90 items, which capture more than 80% of the constructs identified in empirical studies published in the IVA conference 2013-2018. This paper reports on construct validity analysis, specifically convergent and discriminant validity of initial 131 instrument items that involved 532 crowd-workers who were asked to rate human interaction with 14 different ASAs. The analysis included several factor analysis models, and resulted in the selection of 90 items for inclusion of the long version of the ASA questionnaire. In addition, a representative item of each construct or dimension was select to create a 24-item short version of the ASA questionnaire. Whereas the long version is suitable for a comprehensive evaluation of human-ASA interaction, the short version allows quick analysis and description of the interaction with the ASA. To support reporting ASA questionnaire results, we also put forward an ASA chart. The chart provides a quick overview of agent profile.

本研究公开了人工社会智能体（Artificial Social Agent，ASA）问卷研发相关的技术报告、数据集与分析结果。ASA问卷是用于评估人类与人工社会智能体交互的标准化工具，其研发历经多年，汇聚了全球超过100名智能虚拟代理（Intelligent Virtual Agent，IVA）研究者的工作成果。
该问卷包含由90个题项构成的19个测量构念，覆盖了2013-2018年IVA学术会议上发表的实证研究中所确认的80%以上的构念。
本报告针对初始131个问卷题项的结构效度分析展开阐述，具体涵盖聚合效度与区分效度分析；本次分析共招募532名众包工作者，要求其对人类与14种不同人工社会智能体的交互体验进行评分。
本次分析采用了多种因子分析模型，最终筛选出90个题项，构成ASA问卷的长版本。
此外，研究团队从每个测量构念/维度中选取代表性题项，构建了包含24个题项的ASA问卷短版本。
长版本问卷适用于对人类-人工社会智能体交互进行全面评估，而短版本问卷则可快速完成与人工社会智能体交互的分析与描述。
为便于ASA问卷结果的汇报展示，研究团队还提出了ASA可视化图表，该图表可快速呈现智能体的概况信息。