Series 03 / Chapter 04: KV-Cache Arithmetic and Memory

MODEL SERVINGCHAPTER 04 / 10

Cache memory is the admission currency.

Every decoder layer stores a key and value vector for every live token. With grouped-query attention, the cache uses the number of key/value heads rather than the number of query heads. Before sharding and allocator metadata, the approximate bytes for one sequence are:

KV-cache memoryM ≈ 2 × L × S × Hkv × Dh × b

L layers, S cached tokens, Hkv key/value heads, Dh head dimension, b bytes per stored element; the factor two represents keys and values.

For 32 layers, 8 key/value heads, head dimension 128, and two-byte cache values, each token consumes about 128 KiB per sequence. An 8,192-token sequence is therefore about 1 GiB before allocator overhead and any device partitioning. Batch count alone is not a capacity measure: eight short prompts and eight near-limit prompts require radically different cache budgets.

Contiguous allocation reserves a large region for each possible sequence and wastes space when lengths vary. Paged allocation divides the cache into fixed-size blocks, maps logical token positions to physical blocks, and returns blocks as sequences finish. It reduces external fragmentation and lets the scheduler admit work using available blocks rather than worst-case sequence reservations.

Prefix caching can reuse blocks for an identical token prefix, reducing repeated prefill work. Cache identity must include every value that changes model state: model revision, adapter, tokenizer, chat template, positional policy, and relevant multimodal inputs. Tenant boundaries and access policy must prevent one caller from observing or timing private cached prefixes.

Turn model dimensions into an admission decision.

For a decoder with one key and one value vector per layer, cache bytes grow with layers, KV heads, head width, precision, and the number of live token positions. The cache is per sequence, so concurrency multiplies the result. This is why a server can have enough memory for the weights yet fail as soon as several long conversations overlap.

Approximate KV bytes2 x layers x kv_heads x head_dim x bytes_per_element x live_tokens

The factor of two stores both keys and values; allocator metadata and fragmentation add overhead.

As a worked example, 32 layers, 8 KV heads, head width 128, and two-byte cache values consume about 128 KiB per cached token. An 8,192-token sequence therefore approaches 1 GiB before allocator overhead. Ten such requests can consume roughly 10 GiB even though every request uses the same weights.

LeverWhat it savesWhat must be checked

Grouped or multi-query attention

Fewer KV heads per token

Architecture is fixed at training time

Lower-precision cache

Bytes per cached element

Long-context output drift

Paged allocation

Fragmentation and contiguous-allocation pressure

Block size and scheduler overhead

Prefix reuse

Repeated system or document prefixes

Exact matching, isolation, and invalidation

Admission limits

Prevents unsafe oversubscription

Honest maximum output estimates

projected_blocks = blocks(prompt_tokens + max_output_tokens)
if projected_blocks > free_blocks_after_reserve:
    reject_or_queue(request)
else:
    reserve(projected_blocks, request_id)

Reserve against the requested maximum, then return unused blocks as soon as a sequence stops or is cancelled. Monitor free blocks, allocation failures, prefix-cache hit rate, evictions, and bytes per active token. GPU utilization alone cannot reveal a cache-bound server.

Prove the cache model under churn.

Create requests with staggered prompt lengths and random completion times. Some should cancel early; others should grow to their maximum output. Compare projected block use with observed use at every step. A leak appears when finished requests stop generating but free capacity does not return. Fragmentation appears when total free memory exists but the allocator cannot satisfy new sequences.

Test prefix reuse with exact matches, one-token mismatches, tenant boundaries, template revisions, and invalidation after a model update. A high hit rate is not sufficient if cached state crosses isolation boundaries or survives an incompatible template change.

Capacity evidence should include:

bytes per live token and allocator overhead;
free, reserved, active, and evicted blocks;
admission rejects by projected prompt and output;
cache recovery after cancellation and worker drain.