MUSE: In the face of an exercise with multiple solutions, AI tutors impose their preferred method rather than following that of the student

In an environment simulating student-tutor exchanges in programming, LLM leans for brute force, diverts the student from his approach, then remains frozen when a better solution exists.

A good tutor, faced with a problem that admits several valid solutions, relies on the student's strategy before showing him other avenues at the right time. To find out if LLMs used as tutors are capable of doing this, researchers designed MUSE, a controlled environment that simulates conversations between students and tutors on programming exercises with multiple solutions.

The results are not very flattering. AI tutors have marked preferences between solutions that are all correct, with a penchant for brute force approaches. More annoyingly, they tend to redirect the student towards their preferred method instead of accompanying them on their own. Then, once a solution is completed, the behavior is reversed: they remain attached to the current approach while a more effective alternative would deepen understanding, and sometimes switch to a less good option.

The good news, however, is that targeted post-training improves both their ability to follow the student's reasoning and the quality of transitions to better solutions.