By training 800 models with varying proportions of AI-generated text, researchers have shown that a synthetic token can be helpful at times and harmful at others, and have proposed a scaling law that takes this into account.
A team including Mohit Iyyer, John Wieting and Pangram researchers measures the share of IA-generated text in pre-training web data: according to the Pangram detector, it reaches 27.5% of tokens in June 2026 and 31.1% in August. To understand the effect of this contamination, the authors 800 models by varying the ratio between human text and IA text.
A contribution whose sign is reversed
The result depends on the data budget. When a model lacks human text, adding IA text first improves its loss on human text, then the effect quickly returns. For a model already richly nourished with human text, the addition degrades performance from the outset. The researchers propose a law of scale where the value of an AI token can become negative, and which falls on the law of Chinchilla in the absence of synthetic text; it predicts the effect on models up to 3.6 times larger with a reduced error of 41%.
They publish WildAI, a corpus of 83 billion annotated tokens (origin IA, theme, format), as well as 800 models and code.
Read the full article on the original source
Go to the source →