Software Engineering Students in the Face of AI: Three times out of four, they correct an inappropriate answer rather than a false answer

The analysis of 962 exchanges recorded by 145 first-year students shows that supervising an AI is mainly about judging whether its production is suitable for the project, rather than tracking errors.

With generative AI, the work of future software engineers is shifting from production to monitoring what the machine produces. A study published on arXiv examines how beginners exercise this supervision in a real-world setting: 145 first-year students kept a structured "AI diary", totalling 962 exchanges with the tool, including 172 interventions that they themselves reported.

Relevance over accuracy

In 74.4% of these interventions, the problem identified was the adequacy of the response to the engineering task, compared to only 16.9% for an error per se. In other words, students mainly use plausible productions that are poorly adjusted to their needs. To remedy this, they most often modify the result themselves (51% of usable cases) or reformulate their request (28.2%).

The individual analysis also reveals that an answer that is judged to be slightly imperfect is accompanied by less confidence in one's own assessment than an answer judged to be perfect. For the authors, learning to supervise an AI therefore goes far beyond detecting errors: it is an engineering judgment in its own right, which must be taught as such.