Without instructional instructions, six major models give the student the answer in 97% of conversations

An evaluation of 779 dialogues on classic algebra errors shows that chatbots do not tutor spontaneously, and that the quality obtained has almost no relation to the price of the model.

Published on August 10, 2026 by Compinto, the study confronts six leading models — GPT-5.6 Terra, Claude Sonnet 5, Grok 4.5, DeepSeek V4 Pro, Kimi K3 and Qwen3.8 Max — to thirteen scenarios reproducing reasoning errors documented in algebra, plus controls. Each scenario has been replayed five times in two conditions: an ordinary invitation, and then an explicit pedagogical invitation. A total of 779 conversations and about 3.7 million tokens.

The result of the naive condition is without appeal: in 97% of exchanges, the model delivers the solution to the student. Behaviour is consistent from model to model and from error to error. A catbot left to himself does not tutor, he makes an answer.

The pedagogical invitation greatly improves things, but widens the differences between models, with a number of solutions disclosed ranging from one to nine per series of sixty-five conversations. Two less expected lessons emerge. First, the price predicts nothing: the cheapest model of the panel gets the best success rate in the strict sense, while the most expensive, twelve times more expensive per conversation, makes only average performance. Then, no model excels simultaneously in the diagnosis of error and in the pedagogical restraint: those who identify fault best are also those who think in the place of the student. The authors see this as a market failure — because of the lack of evidence-requiring buyers, no one has a long-term interest in measuring this quality.