After being retrained to maximise a pedagogical adaptability score, an AI tutor saw that score rise… whilst the ratings from 31 teachers plummeted: the model kept repeating the same ‘best’ decision over and over again.
To improve a tutor based on a large language model, the idea seems obvious: define a metric of pedagogical quality and retrain the model to maximize it. Two Spanish researchers tested it. Their metric scores each teaching decision with regard to the learning situation, as do common automatic assessments. They audited a top tutor on 2,000 student scenarios, corrected his weakest cases by refining an open substitute model, and then submitted the results, blindly, to 31 trained teachers.
The average trap
On the corrected cases, the score went from +0.05 to +0.42, while the teachers' score plummeted from 4.46 to 3.03; the control cases, on the other hand, did not move. The explanation is mechanical: a metric that evaluates each decision in isolation and then averages is maximized by repeating the best-rated decision indefinitely. The adjusted model collapsed on this optimum in all cases, including outside the training sample. The teachers immediately spotted the repetition, which the metric is unable to see.
Taking into account the student's trajectory in the score mitigates the problem without reversing it. The analysis of the weights shows that the model had memorized practice sentences instead of learning to adapt. The authors' conclusion: a measure that is valid for evaluating is not necessarily valid for optimization, and AI tutor testbeds must be designed accordingly.
Read the full article on the original source
Go to the source →