Banning researchers from using AI to assess the merits of scientific papers doesn’t make much difference to the conclusions the reviewers reach – in large part because many of them ignore the prohibition and use AI anyway. A team led by computer scientists at Microsoft Research ran a large randomised experiment during the 2026 International Conference on Machine Learning (ICML), one of the world’s biggest AI conferences, which took place in Seoul, South Korea in July. Some 24,661 papers were submitted to the conference, requiring oversight from 17,886 reviewers.
When researchers submitted their work, they could choose how it would be reviewed. The first option was to have it assessed by reviewers using a conservative policy that entirely banned the use of large language models (LLMs) to help with reviewing. The other option was to have the assessment performed by reviewers using a permissive policy that allowed the use of LLMs to help understand papers, check related work and polish reviewers’ own writing, but not to judge the merits of a paper or draft a review of the work.
Scientists prefer feedback from ChatGPT to judgement by peers The team led by Microsoft Research scientists discovered that papers reviewed under the two policies had almost identical acceptance rates – 27 per cent under the stricter policy and 26.5 per cent under the permissive one – while average review scores were 3.31 and 3.32 out of 6, respectively. Reviewer confidence was also virtually unchanged. Reviews written under the permissive policy were around 5.5 to 7 per cent longer, and were rated as slightly higher quality by experts, although a reviewer-by-reviewer comparison found no meaningful difference.
The reason for that lack of difference became clear after an anonymous survey of 1486 of the reviewers who took part in the exercise. There, 22.5 per cent of reviewers who were instructed not to use AI admitted to using an LLM anyway. Those who used AI when told not to did so to brainstorm feedback, draft review text, read the papers and summarise their strengths and weaknesses.
In a separate part of the study, the researchers analysed the reviews using Pangram, an AI text detector. They found that only 52.2 per cent of reviews under the conservative policy were classified as fully human-written, compared with 37.0 per cent under the permissive policy – though the researchers warn AI text detectors aren’t perfect. Kim at Microsoft Research.
Extract — continue reading at the source.
