System One models Are Not Better Rerankers
Michael Dinzinger, Kanishka Ghosh Dastidar and Michael Granitzer (University of Passau, Interdisciplinary Transformation University Austria) test whether decision fine-tuning, used to build System One models that answer with a calibrated probability, produces better rerankers than plain reranker fine-tuning of the same backbone.
Ask this paper
Setup. Six Qwen backbones from 0.6B to 27B, each compared as the untuned model asked zero-shot, as decision fine-tunes and as reranker fine-tunes (two trained by the authors), all reranking the same BM25 top-100 on TREC DL 2019 to 2023 and five BEIR collections.
Ranking. Decision fine-tuning never gives the better reranker. From 0.6B to 9B, reranker fine-tuning wins on every backbone by 5 to 29 points nDCG@10, including on zero-shot collections; at 27B the two tie.
No gain over zero-shot. Up to 9B, decision fine-tuning adds nothing over asking the untuned model the same yes or no question.
Calibration. Asked whether a passage answers the query, decision models are better calibrated than published reranker fine-tunes from 4B up, but a slightly different question removes the advantage. Up to 9B, a reranker fine-tune still keeps better passages at a fixed threshold.
Abstract
System One models, or decision models, answer a question about an input with a calibrated probability instead of generated text. Within weeks of the first release, in September 2026, they were used as rerankers. We test whether the decision fine-tuning behind them makes a better reranker than plain reranker fine-tuning of the same model. For six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, we compare the untuned model, asked zero-shot, with its decision fine-tunes and its reranker fine-tunes, two of which we train ourselves. In our work, we draw the conclusion that decision fine-tuning never makes the better reranker. From 0.6B to 9B, reranker fine-tuning wins on every backbone by 5 to 29 points nDCG@10, also on zero-shot collections, and at 27B the two tie. Up to 9B, decision fine-tuning adds nothing over simply asking the untuned model. Moreover, we find that the calibration advantage of decision models depends on the question. Asked whether a passage answers the query, they are better calibrated than the published reranker fine-tunes from 4B on. Asked a slightly different question from prior work, they lose this advantage. Lastly, calibration does not make up for a worse ranking: up to 9B, a reranker fine-tune keeps the better passages at a fixed threshold. If you need a reranker, fine-tune a reranker.