Using the SOCIO Chatbot for UML Modelling: A Second Family of Experiments

CONTENTS OF THIS FILE * Data Analysis * Experimental Design Data Analysis This folder consists of data obtained from the experiments and the materials we use for analysis. Here we provide the following documents: 01-The folder "Class diagrams" This folder contains class diagrams realized by participants in three experiments. 02-Code.R Here we provide the code we use to analyze the raw data of three experiments. With this code, we generate profile-plots, violin-plots, the summaries of descriptive statistics. 03-Familiarity Result.xlsx Here we provide the data we obtained from the familiarity questionnaire. 04-Raw Data.xlsx Here we provide the aggregation of the raw data of the three experiments. 05-Task 1_Ideal Class diagram.png and 06-Task 2_Ideal Class diagram.png Here we provide two ideal class diagrams corresponding to task 1 and task 2. To analyze the response variable effectiveness and the quality of class diagrams, we compare these two ideal class diagrams with each of the class diagrams realized by participants. 07-How to calculate the quality.pdf Here we provide two examples of SOCIO and Creately to illustrate how we compare the ideal class diagram and the class diagram obtained in the experiment. 08-The folder "Thematic Analysis" This folder contains the code and data that we use to generate the thematic results graphs. Experimental Design This folder contains the original materials used in the experimental design process. Here we provide the following documents: 01-Experiment Procedure.pdf Here we provide the experiment procedure in detail. The experiment procedure is identical among the three experiments. 02-Informed Consent.pdf Here we provide the informed consent in detail. We require participants to sign the informed consent before the experiment, stating that they granted us permission to record their data via Telegram. We ensured that the participation of all subjects in this research was completely anonymous, and no information they shared could be electronically traced to them 03-The folder "Questionnaires" This folder contains the familiarity questionnaire we used before the execution of each experiment (i.e., Familiarity Questionnaire.pdf), the SUS questionnaire for Task 1 (i.e., SUS Questionnaire_Task1.pdf) and the SUS questionnaire for Task 2 (i.e., SUS Questionnaire_Task2.pdf). 04-The folder “Tasks” This folder contains the specific content of Task 1 (i.e., Task1.pdf) and Task 2 (i.e., Task2.pdf). 05-The folder “Tutorial Slides” This folder contains the slides we use to give the tutorials of the chatbot SOCIO (i.e., Tutorial_SOCIO V1.pdf) and the web-based online tool Creately (i.e., Tutorial_Creately.pdf).

本文件内容如下：
* 数据分析
* 实验设计

## 数据分析
本文件夹包含实验获取的数据与分析所用材料，现提供如下文件与文件夹：
01. 文件夹「类图（Class diagrams）」：内含三次实验参与者所绘制的类图。
02. Code.R：提供了用于分析三次实验原始数据的代码，借助该代码可生成轮廓图（profile-plots）、小提琴图（violin-plots）以及描述性统计汇总结果。
03. Familiarity Result.xlsx：提供了从熟悉度调查问卷中获取的实验数据。
04. Raw Data.xlsx：提供了三次实验原始数据的聚合集合。
05. Task 1_Ideal Class diagram.png 与 06. Task 2_Ideal Class diagram.png：提供了对应任务1与任务2的两份标准类图。为分析响应变量有效性与类图质量，我们将这两份标准类图分别与参与者绘制的各类图进行对比。
07. How to calculate the quality.pdf：提供了SOCIO与Creately的两个使用示例，用以说明如何对比标准类图与实验中获取的类图。
08. 文件夹「主题分析（Thematic Analysis）」：内含用于生成主题分析结果图表的代码与数据。

## 实验设计
本文件夹包含实验设计流程中使用的原始材料，现提供如下文件与文件夹：
01. Experiment Procedure.pdf：详细阐述了实验流程，三次实验的实验流程完全一致。
02. Informed Consent.pdf：详细提供了知情同意书。实验前要求参与者签署该知情同意书，确认其同意我们通过Telegram记录其实验数据。本研究确保所有受试者的参与完全匿名，其共享的任何信息均无法通过电子手段追溯至其本人。
03. 文件夹「调查问卷（Questionnaires）」：内含每次实验前使用的熟悉度调查问卷（即Familiarity Questionnaire.pdf）、任务1所用的SUS调查问卷（即SUS Questionnaire_Task1.pdf）以及任务2所用的SUS调查问卷（即SUS Questionnaire_Task2.pdf）。
04. 文件夹「任务（Tasks）」：内含任务1的具体内容（即Task1.pdf）与任务2的具体内容（即Task2.pdf）。
05. 文件夹「教程幻灯片（Tutorial Slides）」：内含用于讲解聊天机器人SOCIO的教程幻灯片（即Tutorial_SOCIO V1.pdf）与基于网页的在线工具Creately的教程幻灯片（即Tutorial_Creately.pdf）。