Medusa
Free while signed in. Answers cite the passages they came from.
Medusa accelerates LLM inference by bolting on multiple decoding heads that predict several future tokens in parallel, dramatically reducing decoding steps.
Parallel multi-head decoding: Attaches K extra "Medusa heads" to a base LLM, each trained to predict the k-th next token; a verifier step accepts the longest consistent prefix.
2.2x+ speedup (Medusa-1): Delivers over 2.2x end-to-end inference speedup without quality loss when the heads are trained on top of a frozen backbone.
2.3-3.6x speedup (Medusa-2): Jointly fine-tuning the base model with the heads gives a further bump to 2.3-3.6x speedup while preserving generation quality.
Simpler than speculative decoding: Avoids running a separate draft model, making Medusa cheaper to deploy and easier to reason about than traditional speculative-decoding setups.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack