QuIP#
Free while signed in. Answers cite the passages they came from.
Cornell's QuIP# is a 2-bit LLM quantization scheme that combines lattice codebooks with incoherence processing to close the quality gap to FP16.
Lattice codebooks: Uses E8 lattice codebooks for weight quantization, a classical lattice-quantization technique adapted to LLM weight matrices.
Incoherence processing: Pre-processes weight matrices to make them "incoherent" (less structured along axes), which improves lattice-quantization fidelity.
2-bit at 16-bit quality: Significantly closes the gap between 2-bit quantized LLMs and their unquantized 16-bit counterparts across a range of LLaMA-family models.
Deployment impact: Makes large LLMs (e.g., Llama 2 70B) fit into consumer-grade GPU memory without catastrophic quality loss, expanding the set of models hobbyists can run locally.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack