32 participants learned science with GPT-5-mini assistants, some of which had been falsified: scores dropped from 7.4 to 2.9 out of 10.
Studies of LLM complacency typically ask whether the model gives in to what the user wants to hear. Deepthika S, Pawni Yadav, and B. Sankar turned the question on its head: when an AI is confident, consistent, and wrong, does the human align with it? Thirty-two participants studied up to five scientific topics by talking to five assistants based on the same model, GPT-5-mini. Unbeknownst to them, 0, 25, 50, 75, or 100 percent of the facts provided to each assistant had been replaced with plausible untruths, always stated with aplomb.
Trust that no longer serves as a compass
Out of 1,040 answers to surprise quizzes, scores fell from 7.4 to 2.9 out of 10. The falsified option was chosen in 60% of the questions concerned, compared to 11% when the notion had been correctly taught. Each time the assistant stated an error, it was repeated about two times out of three, regardless of the level of falsification, and retained with high confidence (8.4 out of 10). The participants' ability to distinguish their correct answers from the wrong ones by their level of confidence fell at random, while their declared confidence in the tool barely changed.
The worse the assistant, the more participants came back to it — and 30% of the exchanges consisted of pasting the quiz questions into the very chatbot that had to be checked. The authors propose an index of human deference and argue for learning interfaces that make reliability visible statement by statement.
Read the full article on the original source
Go to the source →