PowerInfer
Free while signed in. Answers cite the passages they came from.
A high-speed LLM inference engine for consumer GPUs that exploits sparse neuron activation patterns to run large models on commodity hardware.
Hot/cold neurons: Analysis shows that a small fraction of "hot" neurons activate on most inputs while the majority of "cold" neurons activate rarely - a power-law pattern across many LLMs.
GPU-CPU hybrid: Hot neurons are preloaded onto the GPU for fast access, while cold neurons live on the CPU and are computed lazily, dramatically reducing GPU memory pressure.
Reduced memory + transfer: This split reduces both GPU memory demand and the CPU-GPU data transfer that typically dominates hybrid inference cost.
11x speedup over llama.cpp: Achieves up to ~11x faster token generation than llama.cpp on a single consumer GPU for OPT-175B-class models - a step-change for local deployment.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack