Scientific research: working with an AI does not produce better ideas according to human evaluators, but an AI judge prefers them

A comparison of human-AI and human-human teams shows that iterative review, not AI, improves research proposals.

AI is increasingly being presented as a research partner. A team of ten researchers wanted to measure what it really is changing, comparing human-AI teams and human-human teams responsible for formulating research ideas, assessed by both humans and LLM, on arXiv, with an analysis of the source of contributions and questionnaires to participants.

Results against the current

First, the proposals resulting from collaboration with AI are not considered better by human evaluators, while the iterative review continuously improves quality. Second observation: IA judges give similar ratings to those of humans but classify ideas differently, and favour AI-assisted proposals without humans seeing any more. It is also difficult to know whether a text has been produced with or without AI, as the productions overlap. Finally, participants felt that AI was more useful after practice, but noted that it was less time-consuming than expected. For the authors, the value of AI in the idea would be less to the quality of ideas than to its influence on their assessment, attribution and perception by researchers.