Experimental data on trial-by-trial AI assistance, feedback, human confidence and decisions during an AI-assisted decision-making chess puzzle task

The data are collected from a human subjects experiment to examine how human confidence in AI and self-confidence change and impact their decision-making during an AI-assisted decision-making task. The experimental task includes three practice and 30 experimental chess puzzle problems, which 100 participants are asked to solve with AI assistance. For each problem, the goal is to make the best next chess move given a board state. The participants first select an independent move (bmove1), receive AI suggestion (aisugg), make a final move (bmove2) , receive feedback on the final move (feedback2), and report their self-confidence (selfconf) and confidence in AI (aiconf). All of these data indicated in the parentheses are recorded in the dataset, as well as some other information including opponent's last move (wmove), top seven moves with the highest evaluation scores (allgoodmoves), ranking of the AI suggestion (multiPV), goodness of the independent move (feedback1), chess board state before opponent's last move (fen before white move), and chess board state before the participants' move (fen before black move). There are two experimental conditions which differ in the order in which the AI performance (i.e., accuracy of AI suggestions) changes: 1) high-performing (80% accuracy) to low-performing AI (20% accuracy) and 2) low-performing (20%) to high-performing (80%) AI. Each CSV datafile ("data#_#") contains each participant's data, where the first # in the filename indicates the participant number and the second # indicates the condition number. For more detailed description of the dataset and the experiment, please refer to the Data In Brief article (publication in process) and the original research article (https://doi.org/10.1016/j.chb.2021.107018). This dataset can be utilized in various domains such as human-computer interaction, psychology, computer science, and team management in engineering/business that seek to understand human cognition and behavior in human-AI collaboration contexts.

本数据集源于一项针对人类受试者的实验，旨在探究人类对人工智能的信心以及自我信心在人工智能辅助决策任务中的变化及其对决策过程的影响。实验任务包括三个练习和30个实验性的国际象棋谜题，100名参与者被要求在人工智能辅助下解决这些谜题。对于每个问题，目标是在给定棋盘状态下做出最佳的下棋步。参与者首先选择一个独立步（bmove1），接收人工智能的建议（aisugg），然后做出最终步（bmove2），并对其最终步获得反馈（feedback2），同时报告他们的自我信心（selfconf）和对人工智能的信心（aiconf）。所有这些数据（括号内所示）均记录在数据集中，以及一些其他信息，包括对手的最后一步（wmove）、评价分数最高的前七步（allgoodmoves）、人工智能建议的排名（multiPV）、独立步的优劣（feedback1）、对手最后一步之前的棋盘状态（fen before white move）以及参与者移动之前的棋盘状态（fen before black move）。实验设有两种条件，区别在于人工智能性能（即建议的准确性）变化的顺序：1）高表现（80%准确性）到低表现人工智能（20%准确性）和2）低表现（20%）到高表现（80%）人工智能。每个CSV数据文件（“data#_#”）包含每位参与者的数据，其中文件名中的第一个#表示参与者编号，第二个#表示条件编号。有关数据集和实验的更详细描述，请参阅《数据简报》文章（待发表）和原始研究文章（https://doi.org/10.1016/j.chb.2021.107018）。该数据集可用于多个领域，如人机交互、心理学、计算机科学以及工程/商业中的团队管理，这些领域寻求理解人类在人类-人工智能协作环境中的认知和行为。