Why Does Post-Training Quantization Work?
Yuxiang Chen, Michael Beyer, Jun Zhu and Jianfei Chen (Tsinghua University and Bosch AI Research) explain why post-training quantization of pretrained LLMs keeps accuracy even though the error from each quantized weight should compound across layers.
Ask this paper
The puzzle: Randomly initialized models accumulate hidden-state error from quantization quickly, while quantized pretrained models accumulate much less and keep downstream performance, despite never being trained with quantization noise.
Mechanism one: The error a layer newly introduces tends to oppose the error it inherits from its input. The two partially cancel, so the full-precision versus quantized discrepancy grows slowly with depth. This interaction develops during pretraining.
Mechanism two: LM-head geometry preferentially preserves the scores and probabilities of high-ranked tokens, which are the model's most confident predictions.
Evidence: Both mechanisms are verified across models and quantization settings by comparing full-precision and quantized forward passes.
Abstract
Post-training quantization compresses large language models (LLMs) by storing their weights at reduced precision, and each quantized weight introduces an error into the hidden states. Naively, these errors should accumulate with depth and corrupt next-token prediction; randomly initialized models accumulate these discrepancies rapidly, whereas quantized pretrained models accumulate much less hidden-state error and largely maintain downstream task performance, even though they were never trained with quantization noise. This raises the question we address: why does post-training quantization work? Comparing full-precision and quantized forward passes, we identify two mechanisms that characterize pretrained quantization robustness. First, the error a layer newly introduces tends to oppose the error it inherits from the layer's input. The two cancel partially such that the discrepancy between full-precision and quantized passes grows slowly. This counteracting residual interaction develops during pretraining. Our quantitative analysis identifies it as a major factor slowing hidden-error growth. Second, LM-head geometry preferentially preserves the scores and probabilities of high-ranked tokens, which typically represent the model's most confident predictions. Together, these mechanisms explain why quantization error that passes through numerous layers can still produce only small output changes, and we verify the findings across models and quantization settings.