Mo' Models, Mo' Problems: How to best select model pools when designing Multi-Agent Systems
Sara Vera Marjanović, Jiacheng Xu and colleagues at NVIDIA evaluate eight ways to choose which open models go into a multi-agent system, across routing, majority vote and LLM-as-judge architectures on hard science benchmarks.
Ask this paper
Oracle gap: Larger heterogeneous pools raise the theoretical oracle ceiling but often lower achieved accuracy below the best single model.
Best strategy: Selecting candidates within a single model family gives the best improvement over a standalone model among the eight strategies (size, accuracy, answer diversity, error diversity and combinations).
Homogeneous baseline: Single-model systems improve with majority vote (29.4% to 32.2% on HLE) while nearly every heterogeneous group declines.
Implication: Adding arbitrary models introduces instability, so model-pool selection should be evaluated as a design choice; current MAS architectures may suit homogeneous pools better.
Abstract
Multi-agent Systems (MAS) combine multiple model outputs to solve complex reasoning tasks. However, despite rapid growth of available open-source models, there is limited research on how to select optimal model candidates out of this massive pool. We systematically evaluate 8 model selection strategies (including model size, accuracy and answer diversity) across before-generation (routing) and after-generation (majority-voting, LLM-as-a-judge) MAS architectures on challenging scientific benchmarks. Our findings show a significant gap between theoretical oracle potential and actual performance: Expanding candidate pool sizes often degrades performance below that of the top performing base-model. We find that candidate selection within a single model family is the strategy that yields the best relative performance over a standalone model. These results demonstrate that adding arbitrary models to a heterogeneous MAS can introduce system instability, highlighting model selection as a critical design choice for multi-agent systems.