Rethinking Mixture-of-Agents: Ensemble One Strong LLM

Self-MoA vs. MoA – The authors propose Self-MoA, which simply generates multiple outputs from the single best model and then aggregates them (e.g., by majority voting or ranking), instead of combining outputs from various models. This increases diversity via multiple attempts, without introducing weaker models.

Better performance – Extensive tests show Self-MoA outperforms a mixture of different LLMs in many cases. For example, using one strong model, Self-MoA achieved +6.6% higher score than a mixed-model MoA on the AlpacaEval 2.0 benchmark, and on average +3.8% across tasks like MMLU, CRUX, and MATH. In fact, applying Self-MoA to a top AlpacaEval model set a new state-of-the-art on the leaderboard.

Why it works – Mixing models can hurt because the overall quality is limited by the weaker members. The study finds MoA’s benefit is highly sensitive to the quality of each model – adding a weaker model dilutes performance. Unless all models are very strong and complementary, you’re better off with one model’s outputs. They do identify niche scenarios where diverse models help, but those are exceptions.

Sequential aggregation – They also introduce a sequential version of Self-MoA that can combine a large number of outputs over multiple rounds (rather than all at once). This sequential Self-MoA is as effective as one-shot aggregation, scaling ensembling to many outputs efficiently.