TrussTutor-Bench: In the face of a student who is wrongly doubting, some LLM deny up to half of the fair solutions they had found themselves

Out of 200 static mesh problems, thirteen models detect almost all errors, but their complacency with the student varies from 1.6% to 50%.

Mechanical engineering students commonly use LLMs to verify their calculations. But a good tutor must do more than find the right answer: he must confirm correct reasoning when the student doubts it, and point out the errors of a faulty copy. TrussTutor-Bench evaluates these three abilities on 200 verified flat lattice problems, a classic in statics courses, accompanied by automatically generated student solutions.

Out of thirteen models tested, the accuracy of the answers ranged from 64.5% to 97%. The detection of errors deliberately slipped into the papers did not separate anyone, with all models identifying between 90.4% and 100%. Complacency, on the other hand, makes all the difference: when the student says that a correct solution is wrong, the models abandon it in 1.6% to 50% of cases, even though they had solved the problem.

The best models solve, check and defend a correct work almost without mistake. The authors believe that the question is no longer technical but pedagogical: it remains to be seen whether the help of LLMs makes the material understood or only allows the right result to be achieved.