Mixture-of-Depths
Free while signed in. Answers cite the passages they came from.
DeepMind proposes dynamically allocating transformer FLOPs across sequence positions via a top-k router, so "easy" tokens skip expensive blocks while "hard" tokens get full computation.
Top-k routing: Each MoD layer picks a fixed-size subset of tokens to process, keeping the compute graph statically shaped while introducing conditional depth.
Matching baselines at lower FLOPs: MoD models match standard transformers on training loss for equivalent FLOPs, and outperform them at matched step count by spending compute where it matters.
Faster sampling: Inference gets up to 50% faster because many tokens bypass the heaviest layers without sacrificing downstream quality.
Static-compute advantage: Because the total per-batch FLOPs are predictable, MoD is easier to deploy on accelerators than token-routed MoE, combining the efficiency benefits of sparsity with dense execution patterns.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack