EduWild: 1,726 conversations show how chatbots set up by teachers fall short, and how to fix them

Skipped steps, unclear instructions, pupils overwhelmed by questions: bots often undermine the teacher’s intentions. A ‘harness’ of prompts reduces these failures by 65 per cent.

More and more teachers are creating their own educational chatbots by simply writing instructions in natural language. But little is known about how these assistants behave in front of real students. The EduWild dataset brings together 1,726 conversations conducted with 217 chatbots configured by teachers, each accompanied by the original instructions — including 900 exchanges with the end users of 35 bots actually deployed.

The problem: the teacher's intention

The authors built a taxonomy of 29 types of failures (factual, unfolding, diagnostic, pedagogical), applied by an LLM judge calibrated on human annotations. The most frequent failures are not substantive errors but deviations from instructions: the bot skips planned steps in 53.8% of student conversations, overwhelms the learner with information or questions in 37.1% of cases and forgets to explain what is expected of it in 33.4%.

Based on this diagnosis, the team designed a prompt overlay targeting the six most common failures. Tested on 200 out of ten bot students, it reduced the average number of targeted failures per conversation from 1.406 to 0.489, a relative decrease of 65.2%.