Students’ understanding of AI is improving, but their critical thinking remains unchanged — and records of their interactions contradict the questionnaire results

In two Master’s-level courses, a teaching assistant who requires students to justify their position before asking questions only improves the self-reported measures. The recorded exchanges are brief, and some are missing.

The idea that a pedagogical AI should support reasoning rather than deliver conclusions is widely circulated; it has rarely been put to the test in a real course. Fatima T. Zahra, Wei Wang, Frances Harper and Jiangen He are working on this in two master's level research methods courses, formalizing their position in the form of a conjecture map and reporting on an initial design cycle.

The device tested imposes a "constraint first" sequence: the student must state and justify a position before the assistant asks questions. Three dimensions were measured before and after — AI culture, critical thinking, metacognitive awareness — in parallel with the recording of interactions. Only AI culture progresses, and this progression focuses on understanding how AI works; critical thinking, awareness, and knowledge remain unchanged.

However, the changes observed relate only to self-reported measures, which allows for a less flattering reading: they could reflect increased confidence rather than increased ability. Interaction traces, on the other hand, paint a different picture — short exchanges, uneven application of the constrained sequence, and missing recordings. The authors draw a methodological recommendation: to base the evaluation of these devices on interaction traces rather than on what students report about themselves.