Table9_Development of an Instrument to Measure Conceptualizations and Competencies About Conversational Agents on the Example of Smart Speakers.pdf

The concept of digital literacy has been introduced as a new cultural technique, which is regarded as essential for successful participation in a (future) digitized world. Regarding the increasing importance of AI, literacy concepts need to be extended to account for AI-related specifics. The easy handling of the systems results in increased usage, contrasting limited conceptualizations (e.g., imagination of future importance) and competencies (e.g., knowledge about functional principles). In reference to voice-based conversational agents as a concrete application of AI, the present paper aims for the development of a measurement to assess the conceptualizations and competencies about conversational agents. In a first step, a theoretical framework of “AI literacy” is transferred to the context of conversational agent literacy. Second, the “conversational agent literacy scale” (short CALS) is developed, constituting the first attempt to measure interindividual differences in the “(il) literate” usage of conversational agents. 29 items were derived, of which 170 participants answered. An explanatory factor analysis identified five factors leading to five subscales to assess CAL: storage and transfer of the smart speaker’s data input; smart speaker’s functional principles; smart speaker’s intelligent functions, learning abilities; smart speaker’s reach and potential; smart speaker’s technological (surrounding) infrastructure. Preliminary insights into construct validity and reliability of CALS showed satisfying results. Third, using the newly developed instrument, a student sample’s CAL was assessed, revealing intermediated values. Remarkably, owning a smart speaker did not lead to higher CAL scores, confirming our basic assumption that usage of systems does not guarantee enlightened conceptualizations and competencies. In sum, the paper contributes to the first insights into the operationalization and understanding of CAL as a specific subdomain of AI-related competencies.

数字素养（digital literacy）的概念作为一种新兴文化技术被提出，被视作成功融入（未来）数字化世界的必要条件。随着人工智能（AI）的重要性与日俱增，素养相关概念亟需拓展以涵盖AI领域的特定属性。系统的便捷操作带来了使用量的攀升，但与之形成鲜明对比的是，人们对其的概念化认知（例如对其未来重要性的预判）与相关能力（例如对功能原理的掌握）仍存在明显局限。本文以基于语音的对话式AI智能体（voice-based conversational agents）这一AI具体应用为研究切入点，旨在开发一套测量工具，用以评估个体对对话式智能体的概念化认知与相关能力。研究分为三个阶段：首先，将“人工智能素养”的理论框架迁移至对话式智能体素养的研究语境中；其次，开发“对话式智能体素养量表”（conversational agent literacy scale，简称CALS），这是学界首次尝试量化个体在对话式智能体使用过程中的“（非）素养”水平差异。研究共生成29个题项，共有170名参与者完成了问卷作答。通过探索性因素分析（explanatory factor analysis）提取出五个因子，据此构建了五个子量表以评估对话式智能体素养，分别为：智能音箱的数据输入存储与传输、智能音箱的功能原理、智能音箱的智能功能与学习能力、智能音箱的应用范围与潜力、智能音箱的技术（周边）基础设施。对该量表的结构效度与信度的初步检验结果显示表现良好。最后，借助新开发的量表对学生样本的对话式智能体素养进行评估，结果显示得分处于中等水平。值得注意的是，拥有智能音箱并未带来更高的素养得分，这验证了我们的核心假设：系统的使用并不等同于具备完善的概念化认知与相关能力。综上，本研究为对话式智能体素养这一AI相关能力子领域的操作化定义与理论理解提供了首批实证见解。