hLLM: Single Pass Decoding for Generative Reranking
Emil Laftchiev and colleagues at Meta Platforms observe that a generative ranker only ever needs to emit N ordinal values, and exploit that narrow output format to decode the whole ranking in a constant number of forward passes.
Ask this paper
The observation is specific and exploitable. Autoregressive decoding spends one sequential forward pass per token, but a ranking is a permutation, and permutation-structured output admits far cheaper decoding than left-to-right generation.
hLLM reads a score matrix off the prefill. A lightweight self-attention head extracts an N by K item-position score matrix from the LLM's prefill hidden states, so no incremental decoding is needed.
The Hungarian algorithm supplies validity by construction. Ordinals are decoded as the optimal bipartite assignment of that matrix, so the output is a valid permutation without any repair step.
28 ms end-to-end, a 64x speed-up, with ranking quality on par with the teacher, using LoRA fine-tuning combined with teacher ranking distillation.
A complete ablation decomposes architecture, training signal and backbone adaptation, and the authors frame the result as a path toward other constant-decode mechanisms for real-time ranking.
Abstract
Large language models (LLMs) achieve state-of-the-art generative ranking quality, but the ranking they produce must be decoded, and autoregressive decoding spends one sequential forward pass per emitted token. We observe that the only tokens a ranker must emit are the $N$ ordinal values naming the items in ranked order, and that this narrow, permutation-structured output format admits decoding strategies which are much more efficient than left-to-right generation. We introduce hLLM (Hungarian LLM), a format-specialized decoding strategy that decodes all $N$ ordinals in $O(1)$ forward passes. hLLM reads an $N \times K$ item-position score matrix off the LLM's prefill hidden states with a lightweight self-attention head, then decodes the ordinals as the optimal bipartite assignment of that matrix via the Hungarian algorithm, yielding a valid permutation by construction rather than by repair. Through a systematic study of training signals and backbone adaptation, we show that LoRA-based fine-tuning combined with teacher ranking distillation reaches 28 ms end-to-end inference, a speed-up of $64\times$ while maintaining ranking quality on par with the teacher. We provide a complete ablation decomposing the contributions of architecture, training signal, and backbone adaptation. Our framework connects generative ranking to combinatorial optimization, opening a path toward other $O(1)$-decode mechanisms for real-time ranking.