Quantization: Reducing Numerical Precision in Neural Network Weights and Activations

Quantization reduces the numerical precision of neural network weights and activations from floating-point formats (FP32/FP16/BF16) to lower-precision integers (INT8 or INT4). The primary goals are reducing memory footprint and increasing inference throughput, enabling deployment of large language models on hardware with limited memory bandwidth.

Precision Formats and Memory

Format	Bits	Range	7B Model Memory
FP32	32	±3.4×10³⁸	28 GB
FP16 / BF16	16	±65,504	14 GB
INT8	8	−128 to 127	7 GB
INT4	4	−8 to 7	3.5 GB
INT2	2	−2 to 1	1.75 GB (unusable quality)

Quantization Methods

Round-to-Nearest (RTN)

Simplest approach: round each weight w to nearest quantization level. Works adequately for INT8; significant accuracy loss at INT4 for models below ~30B parameters.

GPTQ (Frantar et al., 2022)

Layer-wise Hessian-compensated quantization:

1. Compute approximate Hessian H of layer output w.r.t. weights
2. Quantize weights column by column; after rounding weight w_i, update remaining weights: Δw = −(w_q − w) / H_{ii} · H_{i,−i}
3. Achieves INT4 with <1% perplexity increase at 175B scale

AWQ — Activation-Aware Quantization (Lin et al., 2023)

Key insight: ~1% of weights are “salient” — they process activation values with large magnitudes. AWQ scales these salient weights before quantization, preserving their effective precision without storing them separately in FP16.

LLM.int8(): Mixed-Precision Decomposition

Dettmers et al. (2022) discovered that a small fraction of activation dimensions produce extreme outlier values (>6σ), which cannot be accurately represented in INT8.

Component	Precision	Fraction
~99.9% of weight dimensions	INT8	Standard quantization
~0.1% outlier feature dimensions	FP16	Activation magnitudes exceed INT8 range
Matrix multiply output	FP16	Accumulated from INT8 + FP16 streams

This mixed-precision scheme achieves no accuracy degradation at 8-bit while enabling inference on GPUs with 4× less memory than FP32.

KV Cache Quantization

The kv-cache can also be quantized: INT8 KV cache halves the memory required for cached attention states, allowing longer sequences or larger batch sizes within the same GPU memory budget.

See inference-vs-training-compute for broader inference cost context, knowledge-distillation for a complementary compression approach, and kv-cache for the attention memory that quantization also targets.