Evolutionary Model Merge
Sakana AI proposes using evolutionary algorithms to automatically discover effective merges of open-source models, producing strong composite models without any additional training.
Ask this paper
Two search spaces: Evolution operates in parameter space (weight mixing coefficients) and in data-flow space (which layers of which models to route activations through), giving a richer merge vocabulary than hand-crafted recipes.
Cross-domain transfer: The method produces a Japanese LLM with math reasoning by merging unrelated parents - showing that evolution can graft capabilities from one model onto another even across languages.
New SoTA with less compute: The resulting Japanese Math LLM achieves state-of-the-art on Japanese LLM benchmarks, beating models with many more parameters that were never explicitly trained for either language or math.
Collective intelligence: Positions evolutionary merge as an automated composition paradigm that leverages the open-source community's collective work instead of training every capability from scratch.