Out of 4,000 PNAS abstracts rewritten by 23 versions of LLMs, a detector that identifies 99 per cent of the rewrites before a generational change can identify only 3.8 per cent immediately afterwards.
Journals and conferences are increasingly filtering manuscripts to identify texts written with a large language model. But the reliability of these filters is based on tests carried out against a fixed set of models, while the versions used are constantly changing. Two researchers have measured the effect of this renewal.
23 model versions, from June 2023 to August 2026
They matched 4,000 abstracts published in the Proceedings of the National Academy of Sciences before ChatGPT came along with their rewrites by 23 LLM versions from three publishers, and then trained detectors under several maintenance scenarios. Calibrated to incorrectly report 1% of human text, a detector trained on older versions of an editor spotted more than 99% of rewrites just before the most pronounced generation transition, and only 3.8% just after. Differences in vocabulary between versions largely explain these breaks.
In both simulated scenarios, a filter covering all 23 versions either reports one human summary out of eight or lets one out of three rewrites through the most recent version; a commercial detector also missed most of the rewrites of the version following this transition. The authors recommend that the accuracy of a detector be considered provisional and that it be rechecked with each new model output.
Read the full article on the original source
Go to the source →