The Practical Guide to Model Compression with Quantization (INT8, INT4, FP8)

Overview

Model compression by quantization reduces the numeric precision of neural networks to shrink memory footprint, cut latency, and lower energy—often with minimal accuracy loss. This guide explains how quantization works, common design choices, when to use PTQ (post‑training quantization) vs QAT (quantization‑aware training), calibration strategies, LLM‑specific methods, and practical deployment paths in popular runtimes.

Why Quantize?

• Throughput and latency: Integer math (INT8/INT4) maps well to vector/Tensor Cores and DSP instructions.
• Memory: 4× smaller weights when moving from FP32 to INT8; up to 8× at INT4. Smaller models also reduce I/O and cache misses.
• Power and cost: Fewer bits moved/processed lowers energy.
• Portability: Mobile/edge runtimes are heavily optimized for low‑precision kernels.

Trade‑offs: Lower precision can degrade accuracy and numerical stability, especially in layers with outliers or small variances (e.g., LayerNorm, attention softmax). The art is to choose where and how to quantize.

What Changes Mathematically

Quantization maps real values x to a discrete set of integers x_q using a scale (s) and zero‑point (z):

• Quantize: x_q = clamp(round(x / s) + z)
• Dequantize: x̂ = s · (x_q − z)

Design axes:

• Bit‑width: INT8, INT4 for inference; FP16/BF16/FP8 for mixed precision.
• Symmetric vs asymmetric: Symmetric (z = 0) is simple and common for weights; asymmetric can better fit shifted activations.
• Per‑tensor vs per‑channel: Per‑channel scales (especially for conv/linear weights) preserve accuracy better than a single scale.
• Static vs dynamic: Static precomputes activation scales via calibration; dynamic computes them on the fly (common for activations in NLP).
• Weight‑only vs full quantization: LLMs often quantize weights to 4–8 bits while keeping activations higher precision.

Picking an Approach by Model Family

• CNNs/vision: Static INT8 (weights + activations) with per‑channel weight scales is often close to FP32.
• RNNs/transformers (general): Dynamic INT8 activations with per‑channel weight quantization or mixed precision works well.
• Large Language Models (LLMs): Weight‑only 4–8 bit with group‑wise scales; keep LayerNorm/softmax/embeddings and first/last layers in FP16/BF16. Consider GPTQ/AWQ/SmoothQuant.
• Speech/ASR: Mixed precision is common; clip activations aggressively in early layers.

PTQ vs QAT

• PTQ (Post‑Training Quantization): No retraining needed. Use a small calibration set to estimate ranges. Best for tight timelines and when slight accuracy loss is acceptable.
• QAT (Quantization‑Aware Training): Insert fake‑quant ops during training to learn around quantization noise. Best when PTQ drops too much accuracy or for aggressive bit‑widths (INT4/ternary).

Rules of thumb:

• Try PTQ first (static INT8 for CNNs, dynamic/weight‑only for transformers). If the drop >1–2% absolute or business KPIs degrade, switch to QAT or layer‑selective quantization.

Calibration Strategies That Matter

Calibration determines the scale/zero‑point for activations.

• Range estimators: Min‑max (fast, sensitive to outliers), percentile (e.g., 99.9%), MSE/MAE minimization, KL‑divergence (popular in vision), EMAs over batches.
• Granularity: Per‑tensor for activations; per‑channel for weights. For LLMs, group‑wise (e.g., 64/128‑wide groups) balances accuracy and cost.
• Clipping: Learnable or heuristic clipping (e.g., LSQ’s learnable scale or fixed α·σ) reduces outlier impact.

Advanced Techniques for Modern Transformers and LLMs

• SmoothQuant: Shifts activation magnitude into weights by rescaling pairs (weights, activations) to smooth activation distributions, enabling static INT8 activations.
• GPTQ: Second‑order, block‑wise weight quantization that greedily fits quantization errors to minimize output distortion; strong for 4–8 bit weight‑only.
• AWQ: Activation‑aware weight quantization; selects/boosts important channels before quantizing weights to protect salient signal paths.
• RTN/RTDQ baselines: Simple round‑to‑nearest (with per‑channel scale) can be surprisingly competitive for 4–8 bit when combined with mixed precision in sensitive layers.
• NF4/FP4: Non‑uniform 4‑bit formats for training/finetuning memory savings (e.g., QLoRA contexts). Typically used for training/finetuning rather than pure inference engines.

Mixed‑Precision Recipes

• Keep in higher precision: First/last layers, embeddings, LayerNorm, softmax, attention score computation, and small MLP bottlenecks.
• INT8 where safe: Linear/conv weights with per‑channel scales; activations with static scales after careful calibration.
• INT4 for LLM weights: Use group‑wise scales (e.g., group size 64) and test block sensitivity. Revert sensitive blocks to 8/16‑bit if needed.

A Minimal PTQ Example (NumPy‑style intuition)

import numpy as np

def quantize_per_tensor(x, num_bits=8, symmetric=True):
    qmin, qmax = (-(2**(num_bits-1)), 2**(num_bits-1)-1) if symmetric else (0, 2**num_bits - 1)
    if symmetric:
        s = np.max(np.abs(x)) / qmax + 1e-12
        z = 0
    else:
        xmin, xmax = x.min(), x.max()
        s = (xmax - xmin) / (qmax - qmin + 1e-12)
        z = np.round(qmin - xmin / (s + 1e-12))
    x_q = np.clip(np.round(x / s) + z, qmin, qmax)
    x_hat = (x_q - z) * s
    return x_q.astype(np.int8 if num_bits==8 else np.int32), (s, z), x_hat

PyTorch: Quick Wins with Dynamic and Static Quantization

Dynamic quantization (weights INT8; activations quantized at runtime) is a fast baseline for transformer‑style models:

import torch
from torch import nn
from torch.ao.quantization import quantize_dynamic

model = MyTransformerModel().eval()
quantized = quantize_dynamic(
    model,
    {nn.Linear},  # quantize linear layers
    dtype=torch.qint8
)

Static PTQ with FX graph mode (example for a CNN):

import torch
import torch.ao.quantization as tq

model = MyConvNet().eval()
backend = "qnnpack"  # or "fbgemm" on server x86
model.fuse_model()  # fuse conv+bn+relu where available

qconfig = tq.get_default_qconfig(backend)
prepared = tq.quantize_fx.prepare_fx(model, {"": qconfig})

# Calibration with a representative set
with torch.no_grad():
    for images, _ in calib_loader:
        prepared(images)

quantized = tq.quantize_fx.convert_fx(prepared)

ONNX Runtime: Static Quantization Pipeline

from onnxruntime.quantization import CalibrationDataReader, quantize_static, QuantFormat, CalibrationMethod

class ImageDataReader(CalibrationDataReader):
    def __init__(self, images):
        self.enum_data = iter([{input_name: img} for img in images])
    def get_next(self):
        return next(self.enum_data, None)

quantize_static(
    model_input="model.onnx",
    model_output="model_int8.onnx",
    calibration_data_reader=ImageDataReader(calib_samples),
    quant_format=QuantFormat.QDQ,
    calibration_method=CalibrationMethod.Entropy,
    per_channel=True
)

TensorRT: INT8 with a Calibrator (sketch)

import tensorrt as trt

logger = trt.Logger(trt.Logger.INFO)
builder = trt.Builder(logger)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
parser = trt.OnnxParser(network, logger)
with open("model.onnx", "rb") as f:
    parser.parse(f.read())

config = builder.create_builder_config()
config.set_flag(trt.BuilderFlag.INT8)
config.int8_calibrator = MyEntropyCalibrator(calib_batches)
engine = builder.build_engine(network, config)

Calibration Data: How Much Is Enough?

• 200–1,000 samples often suffice for CNNs with entropy/MSE methods.
• For LLMs, a few thousand tokens from domain‑relevant text stabilizes scales; diversify prompts to cover activation ranges.
• Match preprocessing exactly (resize, normalization, tokenization); mismatches cripple calibration.

Sensitivity Analysis and Layer Selection

• Per‑layer error probing: Replace one layer at a time with dequantized FP32 and measure metric recovery.
• Block ranking: For transformers, measure loss/perplexity delta when a block is quantized; keep the top‑K sensitive blocks in higher precision.
• Hessian/curvature proxies: Blocks with larger curvature are more sensitive; use them to prioritize mixed precision.

QAT Essentials

• Fake‑quant modules insert quant/dequant during the forward pass; gradients pass via STE (straight‑through estimator).
• Learnable scales (e.g., LSQ) improve convergence; pair with activation clipping.
• Training recipe: Warm‑up in FP32/BF16 → enable fake‑quant gradually → lower LR and increase weight decay slightly.
• Stop‑grad on zero‑points and constrain scale to positive values for stability.

Minimal QAT loop sketch in PyTorch:

import torch
import torch.ao.quantization as tq

model = MyConvNet().train()
model.fuse_model()
qconfig = tq.get_default_qat_qconfig("qnnpack")
prepared = tq.quantize_fx.prepare_qat_fx(model, {"": qconfig})

for step, (x, y) in enumerate(train_loader):
    loss = criterion(prepared(x), y)
    optimizer.zero_grad(); loss.backward(); optimizer.step()

prepared.eval()
quantized = tq.quantize_fx.convert_fx(prepared)

Common Pitfalls and How to Fix Them

• Activation outliers wreck INT8 ranges → use percentile/MSE calibration, SmoothQuant, or clip with learnable bounds.
• First/last layers lose too much accuracy → keep in FP16/BF16 or INT8 with tighter per‑channel scales.
• LayerNorm/softmax instability → keep in floating point; quantize adjacent linear ops instead.
• Mismatched preprocessing between calibration and inference → standardize pipelines and seed for determinism.
• Hardware kernel gaps → check that your deployment stack actually has optimized kernels for the chosen dtype/layout.

Measuring Success

Track both model quality and system performance:

• Quality: Top‑1/IoU/BLEU/perplexity; domain KPIs.
• Performance: Latency (P50/P90/P99), throughput (images/s or tokens/s), memory (GB), and power (W/req). Combine into cost per inference.
• Regression guardrails: Fail builds if accuracy delta exceeds thresholds (e.g., >0.5% absolute for CNNs, >0.2 perplexity for LLMs—tune per use case).

Deployment Stacks and Fit‑for‑Purpose Notes

• PyTorch ExecuTorch / XNNPACK: Mobile CPU with INT8 kernels; prefer per‑channel weight quant.
• ONNX Runtime: Broad hardware coverage (CPU, GPU, NPUs); QOperator and QDQ flows; easy PTQ.
• TensorRT: GPU‑centric, best for aggressive INT8 pipelines with custom calibrators; also supports FP8/FP16 mixed precision.
• TFLite / Core ML: Mobile/edge focus; static INT8 and select mixed precision; ensure representative datasets.
• OpenVINO / TVM: Strong CPU/VPU optimizations and flexible quantization passes.

A Practical Checklist

1. Define acceptance criteria: max accuracy drop and latency/memory targets.
2. Choose hardware first; align quant formats with supported kernels.
3. Start with PTQ: fuse ops, per‑channel weights, static INT8 activations with robust calibration.
4. If accuracy suffers, add mixed precision for sensitive layers, try SmoothQuant/AWQ/GPTQ for transformers.
5. Escalate to QAT with learnable scales and clipping if needed.
6. Validate across representative datasets and batch sizes; profile P50/P99.
7. Automate CI checks for both metrics and numerics; pin seeds and tool versions.

Glossary (Quick Reference)

• PTQ: Post‑Training Quantization.
• QAT: Quantization‑Aware Training.
• Per‑channel: One scale per output channel of conv/linear.
• Group‑wise: One scale per small channel group (e.g., 64) for LLM weights.
• SmoothQuant: Rescales weights/activations to smooth activation ranges.
• GPTQ/AWQ: Advanced weight‑only quantization for transformers/LLMs.
• LSQ: Learned Step‑size Quantization (learnable scales in QAT).

Closing Thoughts

Quantization is no longer a last‑mile trick—it’s a primary design lever for deploying efficient AI. By combining smart calibration, selective mixed precision, and (when necessary) QAT or modern LLM‑specific methods, you can reach near‑FP accuracy with a fraction of the cost. Start simple, measure often, and iterate layer‑wise until you cross your performance and quality targets.