Branch-Train-MiX (BTX)
Free while signed in. Answers cite the passages they came from.
Meta's BTX produces a single Mixture-of-Experts LLM by first training specialized experts in parallel and then mixing them, sidestepping the high cost of training one big generalist.
Branch: Start from a seed LLM and branch multiple copies, each trained in embarrassingly parallel fashion on a different domain (e.g., code, math, Wikipedia).
Mix: The experts' FFNs are combined into MoE layers of a single model while non-FFN parameters are averaged, producing a unified MoE LLM without additional joint pretraining.
Light fine-tune: Final supervised fine-tuning learns routing and harmonizes the experts, yielding token-level specialization at inference time.
Better accuracy-compute trade-off: BTX matches or beats training-one-generalist and unified-MoE baselines on multi-domain benchmarks at substantially lower total compute, generalizing both Branch-Train-Merge and sparse upcycling.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack