Reinforcement learning teaches nothing new: it chooses, from 1 to 3% of the chips

An alternative without RL achieves comparable performance for about a thousand times less training cost — a few dozen problems and a few minutes on a single GPU.

The conventional wisdom is that reinforcement learning teaches language models new reasoning skills. Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger and Viktor Prasanna argue that RL only redistributes the mass of probability among solutions that the basic model already contains.

Their analysis locates the effect of the RL on only 1 to 3% of token positions, those where the model hesitates. And the token finally promoted is systematically among the top five candidates of the base model. In other words, the model already knows the relevant reasoning paths; it only needs to commit to the right branch at a few critical moments.

An immediate practical consequence

The authors derive ReasonMaxxer, an RL-free method that applies a contrastive loss only to decision points selected by entropy. It achieves performance comparable to the full RL on several model families and several benchmarks, for a training cost that is about three orders of magnitude lower: a few dozen problems and a few minutes on a single GPU.