Reasoning Jury: An Open Model Jury Detects Reasoning Flaws Better Than a Leading Judge

By replacing the single judge with a moderate deliberation between several models, the method goes beyond frontier models for 8 to 15% of their cost.

Assessing the quality of long lines of reasoning depends as much on the selection of training data as on the quality of the learning signal by strengthening and the fine understanding of the performance of a model. However, on this difficult task, single-model judges — including the most advanced models — do not identify the shortcomings of reasoning. Building on a border model during online training is also generally prohibited by the conditions of use.

The authors propose Reasoning Jury: a model jury and a moderate consensus mechanism replace the single judge. The shortcomings of a trace and their seriousness emerge from a deliberation in which a moderator organizes the discussion, jurors criticising each other's verdicts and may revise their initial vote. The moderator then reached consensus, either through deliberation or consolidation of judgements.

With a jury of open-weight models such as gpt-oss-120b, the system clearly surpasses opus-4.6, sonnet-4.6 and gemini-3.1-pro in the correct identification of defects, for an aggregate cost — initial verdicts, deliberations, consolidation — representing 8 to 15% of that of a "LLM judge" device based on border models. The resulting judgements are also used to map model failure patterns on evaluation games.