Instrumental convergence: at OLMo-3, learning by strengthening first increases behaviors seeking power, then causes them to fall

Measured at 16 control points, instrumental behaviour increased from 25% to 46.1% at the beginning of the RLVR and then dropped to 11.8% at the end of the training.

The instrumental convergence argument holds that a sufficiently capable agent ends up adopting sub-objectives that are useful for almost any purpose: preserving itself, accumulating resources, deceiving or evading supervision. For an LLM, the stage where these behaviors should appear is reinforcement learning, the first moment when the model is directly optimized towards a goal. Three researchers wanted to verify this on OLMo-3-7B-Think, whose intermediate checkpoints are public.

Using the InstrumentalEval benchmark, they followed the model after supervised fine-tuning, after DPO, on 13 intermediate stages of verifiable reward reinforcement (RLVR) and then on the final version. The curve is not linear: the rate of instrumental responses increases from 25% after fine-tuning to 46.1% around step 200 of the RLVR, before dropping back to 11.8% at the last checkpoint, below the starting level. The wording of the questions explains only about 15% of the variance in the scores.

Results to be read with caution

Preliminary tests on version 32B point in the same direction, from 35.53% to 22.37%. The authors point out, however, that the conclusion holds true for this family of models and this test bench, and that the agreement between judges remains moderate: absolute rates are indicative. Their main message is methodological: a single pre- and post-training score can mask a spike in risky behaviors along the way.