Experimental data on trial-by-trial AI assistance, feedback, human confidence and decisions during an AI-assisted decision-making chess puzzle task

The data are collected from a human subjects experiment to examine how human confidence in AI and self-confidence change and impact their decision-making during an AI-assisted decision-making task. The experimental task includes three practice and 30 experimental chess puzzle problems, which 100 participants are asked to solve with AI assistance. For each problem, the goal is to make the best next chess move given a board state. The participants first select an independent move (bmove1), receive AI suggestion (aisugg), make a final move (bmove2) , receive feedback on the final move (feedback2), and report their self-confidence (selfconf) and confidence in AI (aiconf). All of these data indicated in the parentheses are recorded in the dataset, as well as some other information including opponent's last move (wmove), top seven moves with the highest evaluation scores (allgoodmoves), ranking of the AI suggestion (multiPV), goodness of the independent move (feedback1), chess board state before opponent's last move (fen before white move), and chess board state before the participants' move (fen before black move). There are two experimental conditions which differ in the order in which the AI performance (i.e., accuracy of AI suggestions) changes: 1) high-performing (80% accuracy) to low-performing AI (20% accuracy) and 2) low-performing (20%) to high-performing (80%) AI. Each CSV datafile ("data#_#") contains each participant's data, where the first # in the filename indicates the participant number and the second # indicates the condition number. For more detailed description of the dataset and the experiment, please refer to the Data In Brief article (publication in process) and the original research article (https://doi.org/10.1016/j.chb.2021.107018). This dataset can be utilized in various domains such as human-computer interaction, psychology, computer science, and team management in engineering/business that seek to understand human cognition and behavior in human-AI collaboration contexts.

本数据集源自一项人类受试者实验，旨在探究在人工智能辅助决策任务中，人类对人工智能的信任度与自身自信心的变化规律，及其对决策行为的影响。本次实验任务包含3道练习棋局与30道正式国际象棋谜题，共100名受试者需在人工智能辅助下完成解题。每道谜题的目标为：给定当前棋盘状态，选出最优的下一步走法。受试者需依次完成以下流程：首先选择独立走法（bmove1），随后接收人工智能建议走法（aisugg），确定最终走法（bmove2），获取最终走法的反馈结果（feedback2），并报告自身自信心（selfconf）与对人工智能的信任度（aiconf）。上述括号内标注的所有变量数据均收录于本数据集，此外还包含其他相关信息：对手的上一步走法（wmove）、评分最高的前七手候选走法（allgoodmoves）、人工智能建议走法的排名（multiPV）、独立走法的优劣反馈（feedback1）、对手上一步走棋前的FEN（Forsyth-Edwards Notation）格式棋盘状态（fen before white move）、受试者走棋前的FEN格式棋盘状态（fen before black move）。本实验设置两种实验条件，二者的人工智能性能（即建议走法的准确率）变化顺序存在差异：1） 先高后低型：人工智能准确率从80%降至20%；2） 先低后高型：人工智能准确率从20%升至80%。每个CSV数据文件均以「data#_#」命名，文件名中第一个#代表受试者编号，第二个#代表实验条件编号。如需了解本数据集与实验的详细细节，请参阅《Data In Brief》（正在出版中）以及原始研究论文（https://doi.org/10.1016/j.chb.2021.107018）。本数据集可应用于多个研究领域，包括人机交互、心理学、计算机科学，以及工程或商业领域的团队管理等，旨在探索人类与人工智能协作场景下的人类认知与行为规律。