The Era of 1-bit LLMs (BitNet b1.58)
Free while signed in. Answers cite the passages they came from.
Microsoft's BitNet b1.58 shows that restricting every weight to the ternary set {-1, 0, 1} can match full-precision FP16 transformers on perplexity and downstream tasks at the same parameter count.
Ternary weights: All linear-layer weights are quantized to {-1, 0, 1}, giving an effective bit-width of log2(3) ≈ 1.58 bits per weight while still permitting meaningful sparsity via the 0 value.
Parity with FP16: At matched parameter and token budgets, BitNet b1.58 matches FP16 baselines in both perplexity and zero-shot accuracy - a result that was not previously demonstrated at 3B+ scale.
Efficiency gains: Dramatic wins in latency, memory footprint, throughput, and energy consumption compared to FP16 transformers, especially at longer sequence lengths.
Hardware co-design: The paper lays out a new scaling law for 1-bit LLMs and argues for specialized hardware tailored to ternary compute - positioning this as a potential inflection point in LLM deployment.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack