UnitBoost: Managing Compound LLM Systems with a Merge Operator, Not a Model
Xing Zhang, Guanghui Wang, Yanwei Cui, Mengdie Flora Wang and Peiyang He (AWS Generative AI Innovation Center) replace the generative manager in a compound LLM system with a defined operator, and show it beats generative managers on three held-out benchmarks.
Ask this paper
What the manager was doing: A meta-agent reads worker outputs, writes the final answer, allocates later calls and decides when to stop. Those are three separate control decisions concentrated in one opaque, order-sensitive model call.
The operator that replaces it: A task-given unit map turns worker outputs into slot-value proposals, a constrained argmax assembles the output, and slots left unfilled or unsupported become an explicit residual that directs the next round.
A guarantee the generative manager cannot offer: Without coupling constraints, unit-wise maximization under the same admission score dominates selection of any complete candidate. The operator is also order-free and records unit provenance.
Numbers: On three held-out benchmarks UnitBoost exceeds the best single candidate chosen with gold labels by 0.060 to 0.195 absolute task-score points, and input-matched generative managers by 0.048 to 0.076. Replacing only the management step improves six compound-system configurations by 0.013 to 0.182. Residual-directed rounds raise FanOutQA cell F1 from 0.4778 to 0.5524.
Where it does not apply: The paper measures three conditions with no available gain: one indivisible unit, unavailable unit identity, and an endpoint that charges per emitted unit. Cross-unit coupling is quantified as a repair cost.
Abstract
Compound LLM systems often solve a coordination problem by adding a higher-level LLM. The resulting meta-agent reads workers' outputs, writes the final answer, allocates later calls, and decides when to stop. It is expressive, but it also concentrates three control decisions in an opaque, order-sensitive model call. We ask whether the manager needs to be generative at all. UnitBoost replaces that model with a defined meta-level operator: a task-given unit map turns worker outputs into slot-value proposals, a constrained argmax assembles the output, and the slots left unfilled or unsupported become an explicit residual for the next round. The operator is order-free, records unit provenance, and gives a simple guarantee: without coupling constraints, unit-wise maximization under the same admission score dominates selection of any complete candidate. On three held-out benchmarks, it exceeds the best single candidate chosen with gold labels by 0.060-0.195 absolute task-score points and input-matched generative managers by 0.048-0.076. Replacing only the management step improves six compound-system configurations by 0.013-0.182. Residual-directed rounds raise FanOutQA cell F1 from 0.4778 to 0.5524; matched controls show that the true residual outperforms random targets and ordinary rereading, while a label-free supply signal flags exhaustion after one unproductive round. The same analysis measures three conditions in which no such gain is available (one indivisible unit, unavailable unit identity, and an endpoint that charges for every emitted unit) and quantifies cross-unit coupling as a repair cost. The manager gives up semantic freedom and gains order invariance, unit provenance, and testable failure conditions.