HACHIMI-II: rather than acting as a pupil, the LLM predicts what that particular pupil will do, including any mistakes

The students simulated through role-play are too good and too similar. By explicitly modelling the learner’s state, HACHIMI-II reproduces their actual mistakes up to five times more often.

Simulated students are used to test intelligent tutors and pedagogical strategies before confronting them with real learners. However, they must succeed and fail in the same places as the student they are imitating. However, the current approach, which consists of asking a large language model to play the role of a student, comes up against a pitfall: the role says nothing about what this student really knows, and it is the competence of the model that fills the gap. As a result, the simulated students are as strong as the model, all look the same, and rarely reproduce real mistakes.

A third-person prediction

Yilin Jiang, Aimin Zhou and their colleagues reverse the approach: simulating a student becomes predicting, from the outside, his or her next action. HACHIMI-II builds a model of the learner from his or her history, infers his or her state in relation to the exercise or exchange in progress, and then translates this state into a set of constraints that frame the response generated.

On three datasets of item responses, the difference between the level of the simulated students and that of the real students is 2.4 to 8.4 times smaller than with the best competing method. On two corpora of tutoring dialogues, the system reproduces the real errors, with the same type of knowledge, the same location and the same mechanism, 1.8 to 5.1 times more often. Ablations attribute most of the gain to the explicit representation of the learner's state.