Supplementary data for the paper 'Turing tests in chess: An experiment revealing the role of human subjectivity'

With the growing capabilities of AI, technology is increasingly able to match or even surpass human performance. In the current study, focused on the game of chess, we investigated whether chess players could distinguish whether they were playing against a human or a computer, and how they achieved this. A total of 24 chess players each played eight 5+0 Blitz games from different starting positions. They played against (1) a human, (2) Maia, a neural network-based chess engine trained to play in a human-like manner, (3) Stockfish 16, the best chess engine available, downgraded to play at a lower level, and (4) Stockfish 16 at its maximal level. The opponent’s move time was fixed at 10 seconds. During the game, participants verbalized their thoughts, and after each game, they indicated by means of a questionnaire whether they thought they had played against a human or a machine and if there were particular moves that revealed the nature of the opponent. The results showed that Stockfish at the highest level was usually correctly identified as an engine, while Maia was often incorrectly identified as a human. The moves of the downgraded Stockfish were relatively often labeled as ‘strange’ by the participants. In conclusion, the Turing test, as applied here in a domain where computers can perform superhumanly, is essentially a test of whether the chess computer can devise suboptimal moves that correspond to human moves, and not necessarily a test of computer intelligence.

随着人工智能（AI）能力的持续提升，技术系统已愈发能够比肩甚至超越人类的表现水平。本研究以国际象棋为核心研究领域，旨在探究国际象棋棋手能否分辨对局对手为人类还是计算机，并阐明其实现辨别过程的内在机制。共有24名国际象棋棋手参与实验，每名棋手需从不同初始布局出发，完成8局5+0超快棋（Blitz）对局。本次实验的对局对手分为以下四类：(1) 人类棋手；(2) Maia，一款基于神经网络训练、旨在模拟人类行棋风格的国际象棋引擎；(3) Stockfish 16，当前顶尖的国际象棋引擎，但被降档至较低的运行等级；(4) 全开算力模式下的Stockfish 16。所有对局中，对手的每步思考时长均固定为10秒。对局过程中，受试者需口头表述自身的思考过程；每局对局结束后，受试者需通过问卷表明自己认为本次对局的对手是人类还是计算机，并指出是否存在某步特定棋步暴露了对手的真实身份。实验结果显示，全开算力的Stockfish 16通常可被受试者准确识别为计算机引擎，而Maia则常被误判为人类棋手。降档后的Stockfish 16所走出的棋步，相对更频繁地被受试者标注为『怪异』。综上，本研究在计算机已具备超人类表现的国际象棋领域中应用图灵测试（Turing test）后发现，该测试本质上是验证国际象棋计算机能否生成与人类行棋风格相符的次优走法，而非用于检验计算机通用智能的测试。