The F-ICL test bench compares the induction in the context of the models to an exact computable standard: the accuracy of the responses masks an inductive bias that the scale does not correct.
Most in-context learning assessments do not have any ground truth about what the correct inductive inference would be: we measure whether the answer is correct, not whether the reasoning was correct. In an article submitted on August 3, 2026, Hector Zenil and Luan Ozelim propose a test bed that provides this reference — F-ICL, built around an optimal Bayesian standard that can be computed exactly. Each task is matched to its bitwise complement, on which the optimum obtains exactly the same score: any gap between a task and its twin therefore isolates the inductive bias specific to the model.
The study involves 37 open models, ranging from 0.8 to 675 billion parameters, supplemented by state-of-the-art systems from four laboratories. The models respond correctly to 92% of the queries. But 45 of the 46 systems produce distributions further from the optimum than a simple reference, and the authors note 6,545 transitions from a resolved state to an unresolved state compared to 13,702 gains — a sign of a non-monotonic update.
The deviation from the optimum is not predicted by precision (Spearman's correlation ρ = −0.19, p = 0.21) and does not close with the size of the model. More troubling: the gains from post-training widen it. In other words, what we are optimizing today improves the final response without bringing the machine closer to a correct inference.
Read the full article on the original source
Go to the source →