Mutation Testing for Task-Oriented Chatbots: Dataset

Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots. To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.对话智能体（Conversational Agent）或称聊天机器人（Chatbot）正日益被用于通过自然语言交互获取各类服务。尽管诸如ChatGPT这类开放域聊天机器人（open-domain chatbot）可围绕任意主题展开对话，但本研究聚焦的任务导向型聊天机器人（task-oriented chatbot）则专为特定任务设计，例如航班预订、客户咨询或预约安排。与其他软件一样，任务导向型聊天机器人也需要经过严格测试，通常通过定义并执行测试场景（即用户与聊天机器人的交互序列）来完成。然而当前尚缺乏可量化此类测试场景完整性与有效性的方法，这可能导致测试质量低下，进而产出存在缺陷的聊天机器人。 为填补这一研究空白，本文提出将突变测试（Mutation Testing，MuT）适配至任务导向型聊天机器人的测试场景。为此，本文引入了一系列用于模拟聊天机器人设计缺陷的突变算子，搭建了可对采用异构技术构建的聊天机器人执行突变测试的架构，并将该方案实现为一款Eclipse插件（Eclipse plugin）。此外，本文在开源聊天机器人上评估了所提方法的适用性、有效性与效率，取得了较为理想的实验结果。