Mathematical Demonstration Courses: Each hundred messages sent to the chatbot costs three points on examination

Of 153 students in evidence courses, early access to a chatbot improved homework but not the exam, and intensive use was accompanied by lower grades.

A team of 21 researchers followed 153 undergraduate students enrolled in a demonstration-based math course. Randomly drawn, they were given access to an AI help system either early or later in the semester. This system included two tools: a conversational chatbot and an assignment feedback interface, which only provides feedback after the student has submitted their own attempt at proof.

Better homework, not better exams

As long as only the first group had access to it, their assignments scored an average of 2.56 points higher, with no significant difference on the exam that followed. Once the tool was open to everyone, every 100 messages sent to the chatbot was associated with a lower exam score of 3.02 points. A mediation analysis links this phenomenon to self-confidence: students who were most doubtful about their abilities used the chatbot more and performed worse, at a rate of 0.62 points lost for every point of confidence decreased.

The homework feedback interface was not linked to lower grades. The authors see this as a sign that students were entrusting their reasoning to the chatbot rather than working on it, and advise against this format for tutoring, in favor of a tutor integrated into the course activities.