QuIP#
Cornell's QuIP# is a 2-bit LLM quantization scheme that combines lattice codebooks with incoherence processing to close the quality gap to FP16.
Ask this paper
Lattice codebooks: Uses E8 lattice codebooks for weight quantization, a classical lattice-quantization technique adapted to LLM weight matrices.
Incoherence processing: Pre-processes weight matrices to make them "incoherent" (less structured along axes), which improves lattice-quantization fidelity.
2-bit at 16-bit quality: Significantly closes the gap between 2-bit quantized LLMs and their unquantized 16-bit counterparts across a range of LLaMA-family models.
Deployment impact: Makes large LLMs (e.g., Llama 2 70B) fit into consumer-grade GPU memory without catastrophic quality loss, expanding the set of models hobbyists can run locally.