AI-based marking of assignments: out of 761 papers from three British universities, no single model or prompt proved universally applicable

The three models tested all tend to produce average marks, over-rating weaker papers and under-rating the best ones – a common error that the aggregation process does not correct.

A team led by Giulio Corsi and Alexandru Marcoci compared three major language models, alone and in combination, with 761 undergraduate psychology assignments from three universities in the United Kingdom. Unlike usual assessments that are limited to one model, one prompt and one institution, the study measures accuracy, prompt sensitivity, error independence and the ability to generalize from one context to another.

Compressed notes towards the center

The agreement between human grades and model grades varies greatly, and no combination of model and prompt is optimal in the three universities. All models tighten grades around a common central tendency: the weakest papers are overgraded, the best undergraded. Since this bias is shared, combining the models does not erase it. Automatic grades also reward the linguistic characteristics of texts more than do human graders.

The authors conclude that current models do not have an evaluation capacity that can be transferred from one context to another, and that validation conducted in a single institution overestimates their maturity for deployment.