XAI-FUNGI: Dataset from the user study on comprehensibility of XAI algorithms

XAI-FUNGI: Dataset from the user study on comprehensibility of XAI algorithms We present the dataset which was created during a user study on evaluation of explainability of artificial intelligence (AI) at the Jagielloninan University as a collaborative work of computer science (GEIST team) and information sciences research groups. The main goal of the research was to explore effective explanations of AI model patterns to diverse audiences. The dataset contains material collected from 39 participants during the interviews conducted by the Information Sciences research group. The participants were recruited from 149 candidates to form three groups that represented domain experts in the field of mycology (DE), students with data science and visualization background (IT) and students from social sciences and humanities (SSH). Each group was given an explanation of a machine learning model trained to predict edible and non-edible mushrooms and asked to interpret the explanations and answer various questions during the interview. The machine learning model and explanations for its decision were prepared by the computer science research team. The resulting dataset was constructed from the surveys obtained from the candidates, anonymized transcripts of the interviews, the results from thematic analysis, and original explanations with modifications suggested by the participants. The dataset is complemented with the source code allowing one to reproduce the initial machine leaning model and explanations. The general structure of the dataset is described in the following table. The files that contain in their names [RR]_[SS]_[NN] contain the individual results obtained from particular participant. The meaning of the prefix is as follows: RR - initials of the researcher conducting the interview, SS - type of the participant (DE for domain expert, SSH for social sciences and humanities students, or IT for computer science students), NN - number of the participant File Description SURVEY.csv The results from a survey that was filled by 149 participants out of which 39 were selected to form a final group of particiapnts. CODEBOOK.csv The codebook used in thematic analysis and MAXQDA coding QUESTIONS.csv List of questions that the participants were asked during interviews. SLIDES.csv List of slides used in the study with their interpretation and reference to MAXQDA themes and VISUAL_MODIFICATIONS tables. MAXQDA_SUMMARY.csv Summary of thematic analysis performed with codes used in CODEBOOK for each participant PROBLEMS.csv List of problems that participants were asked to solve during interviews. They correspond to three instances from the dataset that the participants had to classify using knowledge gained from explanations. PROBLEMS_RESPONSES.csv The responses to the problems for each participant to the problems listed in PROBLEMS.csv VISUALIZATION_MODIFICATIONS.csv Information on how the order of the slides was modified by the participant, which slides (explanations) were removed, and what kind of additional explanation was suggested. ORIGINAL_VISUZALIZATIONS.pdf The PDF file containing the visualization of explanations presented to the participants during the interviews VISUALIZATION_MODIFICATIONS.zip The PDF file containing the original slides from ORIGINAL_VISUZALIZATIONS.pdf with the modifications suggested by the participant. Each file is a PDF file named with the participant ID, i.e. [RR]_[SS]_[NN].pdf TRANSCRIPTS.zip The anonymized transcripts of interviews for each given participant, zipped into one archive. Each transcript is named after the particiapnt ID, i.e. [RR]_[SS]_[NN].csv and contains text tagged with slide number that it related to, question number from QUESTIONS.csv, and problem number from PROBLEMS.csv. The detailed structure of the files presented in the previous Table is given in the Technical info section. The source code used to train ML model and to generate explanations is available on GitlabXAI-FUNGI：可解释人工智能（Explainable AI, XAI）算法可理解性用户研究数据集 本数据集由雅盖隆大学计算机科学（GEIST团队）与信息科学研究团队合作开展的人工智能（Artificial Intelligence, AI）可解释性用户研究项目所构建，旨在探索针对不同受众的AI模型模式有效解释方法。 该数据集包含信息科学研究团队开展的访谈中收集的39名参与者的相关资料。研究人员从149名候选者中招募参与者，将其分为三组：真菌学领域专家（Domain Expert, DE）、具备数据科学与可视化背景的计算机科学相关专业学生（IT）以及社会科学与人文科学（Social Sciences and Humanities, SSH）学生。每组参与者均需接收针对训练完成的可食用/不可食用蘑菇预测机器学习模型的解释，并在访谈中解读该解释并回答各类问题。该机器学习模型及其决策解释由计算机科学研究团队完成。 本数据集由参与者的调查问卷、匿名化访谈转录文本、主题分析结果以及经参与者提出修改建议的原始解释材料构建而成，并附带可复现初始机器学习模型与解释的源代码。 数据集的整体结构见下表。文件名包含[RR]_[SS]_[NN]的文件为单名参与者的个体结果，各前缀含义如下： - RR：开展访谈的研究员姓名首字母 - SS：参与者类型（DE代表真菌学领域专家，SSH代表社会科学与人文科学学生，IT代表计算机科学相关专业学生） - NN：参与者编号 以下为各文件的详细说明： | 文件名 | 描述 | | --- | --- | | SURVEY.csv | 面向149名参与者发放的调查问卷结果，其中39人入选最终参与组 | | CODEBOOK.csv | 主题分析与MAXQDA编码所使用的编码手册 | | QUESTIONS.csv | 访谈中向参与者提出的问题清单 | | SLIDES.csv | 研究中使用的幻灯片清单，附带解读内容及与MAXQDA主题和VISUAL_MODIFICATIONS表的关联说明 | | MAXQDA_SUMMARY.csv | 针对每位参与者、使用CODEBOOK中编码开展的主题分析汇总结果 | | PROBLEMS.csv | 访谈中要求参与者完成的任务清单，对应3个需参与者借助解释获取的知识完成分类的数据集实例 | | PROBLEMS_RESPONSES.csv | 参与者针对PROBLEMS.csv中所列任务的应答结果 | | VISUALIZATION_MODIFICATIONS.csv | 关于参与者修改幻灯片顺序、移除的解释幻灯片以及建议添加的额外解释的相关信息 | | ORIGINAL_VISUALIZATIONS.pdf | 访谈中向参与者展示的解释可视化内容的PDF文件 | | VISUALIZATION_MODIFICATIONS.zip | 包含ORIGINAL_VISUALIZATIONS.pdf原始幻灯片并附带参与者修改建议的压缩包，每个文件为以参与者ID命名的PDF文件，格式为[RR]_[SS]_[NN].pdf | | TRANSCRIPTS.zip | 每位参与者的匿名化访谈转录文本压缩归档，每份转录文本以参与者ID命名，格式为[RR]_[SS]_[NN].csv，其中包含与幻灯片编号、QUESTIONS.csv中的问题编号以及PROBLEMS.csv中的任务编号对应的标注文本 | 各文件的详细结构参见技术信息部分。用于训练机器学习模型并生成解释的源代码已上传至Gitlab平台。