Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
Yunpeng Ba and colleagues (Huawei Noah's Ark Lab, City University of Hong Kong) explain when Evolution Strategies beat GRPO for LLM reasoning, tying the advantage to reasoning coverage rather than to raw reward.
Ask this paper
Coverage, not accuracy, is the mechanism: ES leads to broader reasoning coverage and therefore better exploits the pretrained model's latent capability. The theoretical hook is that verifier-projected Jensen-Shannon diversity across the ES population predicts higher Pass@K.
GRPO's entropy collapse is the contrast: GRPO exhibits entropy collapse during training. ES improves Pass@1 while also reaching higher Pass@K, which is the combination that entropy collapse normally makes impossible.
Memory-efficient post-training: ES avoids storing gradients, which is why it re-entered the conversation for LLM post-training in the first place. This paper supplies the missing account of when that tradeoff is favorable.
Why it matters: Pass@K is the metric that predicts downstream RL headroom. A method that raises Pass@1 without collapsing Pass@K is directly useful to anyone building a reasoning post-training stack.
Abstract
Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES over GRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show that verifier-projected Jensen-Shannon diversity across the ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.