As artificial intelligence (AI) becomes increasingly embedded in daily life, ascertaining whether an agent is human is critical. Here we systematically benchmark AI’s ability to imitate humans in three language tasks ( ing, word association and conversation) and three vision tasks (colour estimation, object detection and attention prediction), collecting data from n = 636 humans and 37 AI agents. We also conducted 72,191 Turing-like tests with 1,916 human judges and 10 AI judges.
Current AIs are approaching the ability to convincingly impersonate humans and deceive human judges in both language and vision. Even simple AI judges outperformed humans in distinguishing AI from human responses. Imitation ability showed minimal correlation with conventional AI performance metrics, suggesting that passing as human is an important independent evaluation criterion.
The large-scale Turing-like datasets and metrics introduced here offer valuable benchmarks for assessing human-likeness in AI and highlight the importance of rigorous, quantitative imitation tests for AI development. Current language and computer vision models excel in a wide range of tasks, such as ing1,2,3,4,5, text generation6,7,8,9,10,11,12, object recognition5,13,14 and attention prediction15,16,17. Developing state-of-the-art algorithms goes hand-in-hand with the creation of better and more precise ways of assessing their performance.
The performance of artificial intelligence (AI) algorithms is often defined by comparing their outputs against human ground-truth annotations. Such metrics are particularly prevalent in computer vision, such as in object recognition tasks. Additionally, several metrics have been adapted from natural language processing tasks to ing, such as BLEU18, THUMB19 and METEOR20 (see also ref. 21).
Generative AI models are notoriously difficult to evaluate due to the inherent ambiguities of creating novel content22. Human evaluators are often recruited to assess the quality of sentiment, semantic relevance, reasoning abilities or emotional valence on text generated by large language models9,22,23,24,25. The Turing test, also known as the imitation game, was proposed by Alan Turing in 1950 as a way of assessing a machine’s ability to exhibit intelligent behaviours indistinguishable from those shown by humans26.
In the imitation game, a machine tries to pass as a human during a conversation, and a human judge determines whether they are interacting with a human agent or a machine26 (Fig. 1a). Since its inception, the Turing test has generated extensive controversy and discussion. Several notable arguments include Searle’s Chinese Room thought experiment27, Block’s behaviourism28, Harnad’s Total Turing Test29, Watt’s Inverted Turing Test30, Damassino’s Questioning Turing Test31 and Sejnowski’s Reverse Turing Test32.
Extract — continue reading at the source.
