Supplementary data for the paper 'Turing tests in chess: An experiment revealing the role of human subjectivity'
CC BY 4.0
With the growing capabilities of AI, technology is increasingly able to match or even surpass human performance. In the current study, focused on the game of chess, we investigated whether chess players could distinguish whether they were playing against a human or a computer, and how they achieved this. A total of 24 chess players each played eight 5+0 Blitz games from different starting positions. They played against (1) a human, (2) Maia, a neural network-based chess engine trained to play in a human-like manner, (3) Stockfish 16, the best chess engine available, downgraded to play at a lower level, and (4) Stockfish 16 at its maximal level. The opponent’s move time was fixed at 10 seconds. During the game, participants verbalized their thoughts, and after each game, they indicated by means of a questionnaire whether they thought they had played against a human or a machine and if there were particular moves that revealed the nature of the opponent. The results showed that Stockfish at the highest level was usually correctly identified as an engine, while Maia was often incorrectly identified as a human. The moves of the downgraded Stockfish were relatively often labeled as ‘strange’ by the participants. In conclusion, the Turing test, as applied here in a domain where computers can perform superhumanly, is essentially a test of whether the chess computer can devise suboptimal moves that correspond to human moves, and not necessarily a test of computer intelligence.
- 2024-09-30 first online, published, posted
DATA
- 4,460 bytesMD5:
112ba71fddad6dadccea433fbef701e6readme.txt - 366,306 bytesMD5:
9ffe3d78775cffb552c971bc0105ee8a@Edf2Mat.zip - 35,706,384,998 bytesMD5:
b5a37da32169096db9e83c869e955d39Data.zip - 1,942,971 bytesMD5:
cd064535df7c2f02a4ed5fccb9f6c2bcExperiment material.zip - 1,502,610,615 bytesMD5:
9fc77c6322af40a1cc6dcc46d712f598Files for Matlab.zip - 48,085 bytesMD5:
d33ed8e9efb19ebc72e9f2ed252983f6read_and_process.m -
download all files (zip)
37,211,357,435 bytes unzipped
