Maths with a chatbot: high school students especially applicants, but those looking to understand along the way succeed better without AI

Out of 112 students using Mistral, exchanges are dominated by requests; However, the evolution of their demands towards understanding predicts their results once AI is removed.

Does a student who works with a chatbot remain an actor in his or her learning? Rania Abdelghani, Peter Kaiser and Kou Murayama analysed 880 exchanges of 112 secondary school students solving mathematical modelling exercises with a generalist model, Mistral. They identified signs of self-regulation (planning, controlling, evaluating), strategies for asking for help and vigilance in the face of responses.

The overall finding is mixed: students mostly make requests, and rarely plan or check, even if a poor quality response from the AI pushes them to evaluate more. As the conversation progresses, self-regulatory behaviors improve a little, but requests become less comprehensive. The key result is elsewhere: the trajectory of requests matters. Students whose questions evolve into comprehension perform better on the final test taken without AI, beyond what their initial level and overall habits predicted. The authors therefore invite us to observe the course of the exchanges rather than isolated types of queries, and publish their anonymized data.