Detecting AI-generated code: out of 30,000 student answers, a similarity to model solutions proves almost nothing

A study of three introductory Python courses shows that the models converge towards the same solutions, and that the students’ solutions also converge towards them, with or without AI.

To identify the use of AI in programming assignments, many teachers compare copies with solutions generated by language models. A team of researchers tested this method on 29,970 students from ten Python TPs, in 2021, 2023 and 2025, faced with 90,000 solutions produced after the fact by three leading models. The item was filed on arXiv on October 1.

Solutions that naturally converge

Models usually produce a correct code and, for most exercises, very similar implementations. Copies of recent promotions are more similar to these references, including those that pass all hidden tests. On the functions very framed by the statement, the models are limited to a few identical syntactic trees, and the diversity of forms rendered by students also decreases from year to year; more open functions remain varied on both sides. Most of the matches detected by the MOSS tool are also very short.

The authors recommend that a bank of solutions be generated before publishing a paper to identify and rewrite exercises on which AI converges to require testing, reasoning or intermediate steps. According to them, the method allows to follow a progression at the scale of a promotion, but accusing a specific student would require other evidence, such as its prompts, successive versions or its own statement.