Quantization changes memory traffic and the output distribution.
Reducing precision can make a model fit, allow more cache blocks, and reduce bandwidth per generated token. Weight-only quantization primarily reduces the cost of reading weights and is often attractive for decode. Weight-and-activation quantization can accelerate more operations but demands a supported kernel path and careful calibration. The KV cache can be stored at lower precision independently of the weights.
Smaller checkpoint and lower read bandwidth
Logit drift, task quality, kernel coverage
Faster matrix operations and less temporary memory
Outlier handling and calibration stability
More concurrent or longer sequences
Long-context and generation-quality drift
Memory reduction is not the same as latency reduction. Dequantization overhead, unsupported layers, small batch shapes, or frequent format conversions can erase the gain. Report the exact scheme, group size, scales, calibration set, fallback operations, and kernel implementation rather than a label such as “4-bit.”
Test parity at three levels: compare logits on fixed prefixes, compare deterministic outputs under fixed seeds and settings, then rerun behavioral evaluations. A small average logit error can still alter a near-tied token choice and cause a long generated sequence to diverge.
Quantization is an artifact transformation, not a filename suffix.
A fine-tuned model may exist as base weights plus a small adapter, as merged full-precision weights, or as a runtime-specific quantized artifact. The serving engine determines which form it can load. Portable local runtimes commonly use a compact container such as GGUF; accelerator servers may load merged weights, separate adapters, or engine-specific quantization. Preserve the tokenizer and template beside every form.
Small and easy to version
Base-model identity and runtime adapter support
Simple loading and broad compatibility
Large export memory and irreversible lineage mistakes
Lower weight bandwidth, often useful for decode
Dequantization overhead and unsupported layers
Can use low-precision matrix hardware
Calibration, outliers, and kernel availability
More context or concurrency
Long-horizon generation drift
Names such as 4-bit, FP8, or FP4 are incomplete descriptions. Report group size, scale format, calibration data, tensors excluded from quantization, fallback kernels, and target hardware. A format accelerated on one GPU generation may silently fall back or become slower on another.
export -> reload in target runtime -> render identical prompt
-> compare token IDs -> compare fixed-prefix logits
-> compare deterministic generations -> run task slices
-> measure memory, TTFT, decode rate, and output lengthTemplate parity is part of this test. Exported models that become repetitive or fail to stop are often suffering from a conversation-template, EOS, BOS, or turn-separator mismatch rather than damaged weights. Compare the rendered prompt and token IDs before blaming quantization.
Keep the unquantized or higher-precision artifact as the semantic reference. The quantized release is acceptable only when its speed and memory gain are measured beside output drift on the tasks and sequence lengths users care about.
Validate numerical changes at increasing semantic distance.
Begin with tensor-level checks: load every expected parameter, confirm shapes, and record which kernels run at which precision. Next compare logits on fixed prefixes. Then compare deterministic generation under identical templates and stop rules. Finally run behavioral slices, long-context tasks, tool calling, and adversarial prompts.
Sequence divergence is expected when near-tied logits swap order, so exact generated-text equality is too strict for every prompt. Instead pair numerical distances with task outcomes, calibration, refusal behavior, output length, and failure examples. The acceptance threshold should follow user risk, not one universal cosine score.
Keep conversion logs and calibration-set hashes with the artifact. Without them, a quantized file cannot be regenerated or audited when a later runtime produces different results.