Series 03 / Chapter 09: Capacity Planning and Reliable Overload

MODEL SERVINGCHAPTER 09 / 10

Overload must be a designed state.

Capacity depends on the joint distribution of prompt length, output length, arrival rate, concurrency, cache reuse, and sampling settings. Request count is insufficient because one 100-token answer and one 10,000-token answer occupy the decode scheduler for very different durations. Maintain separate prompt-token and generated-token budgets, then estimate memory before admission.

Concurrency sanity checkaverage in-flight requests ≈ arrival rate × average service time

Use measured distributions and tail service times for provisioning; this relation is only a first-order check.

Admission control should reject or defer work before the device reaches an unrecoverable memory state. Bound queue length, prompt tokens, generated tokens, live cache blocks, and per-tenant concurrency. Apply deadlines and propagate cancellation. Use backpressure rather than accepting unlimited work and converting every request into a timeout.

ControlFailure it contains

token-aware admission

Cache exhaustion and oversubscribed decode steps

queue deadline + maximum depth

Unbounded tail latency during bursts

cancellation propagation

Compute spent after clients disconnect

worker drain + warmup

Requests routed to cold or terminating replicas

revision-aware routing

Mixed tokenizer, adapter, or engine contracts

bounded retry policy

Duplicate work and retry storms

Autoscaling on accelerator utilization alone is late and ambiguous: a server can be memory-full with modest arithmetic use, or compute-busy while the queue is healthy. Combine queue delay, admitted token work, cache occupancy, active sequences, deadline misses, and load per replica. Keep warm capacity when model loading and compilation take longer than the service’s recovery objective.

Fail closed on contract mismatch. A replica with the wrong tokenizer, adapter, numerical engine, or model revision should not receive traffic merely because its health endpoint responds. Readiness should include a fixed prompt probe, expected token IDs, and a small deterministic output or logit checksum.

Troubleshoot inference from the boundary inward.

Serving failures are often misdiagnosed as weak model behavior. Start by comparing the exported artifact with the reference runtime on one fixed prompt. If token IDs differ, inspect the tokenizer and chat template. If tokens match but generation never stops, inspect EOS and stop rules. If deterministic logits drift, inspect adapter loading, weight conversion, numerical format, or unsupported kernel fallbacks.

SymptomFirst checksDo not assume

Repetition or gibberish

Template, EOS/BOS, turn separators, sampler

The fine-tune is corrupt

OOM during export

Merge strategy, peak-memory limit, shard size

The final artifact cannot fit at runtime

OOM under traffic

KV projection, output limits, fragmentation, reserve

Weight memory is the only memory

Low throughput

Batch shape, kernel fallback, host transfer, collectives

Lower precision is automatically faster

Malformed tool calls

Template, injected system text, schema dialect, parser

The endpoint is semantically compatible

Tail-latency spike

Queue depth, long prefill, cache eviction, retries

The model kernel slowed down

Overload should degrade through designed states: queue briefly, reject before memory exhaustion, propagate deadlines, cancel abandoned work, and preserve a reserve for health probes and high-priority traffic. Unlimited acceptance converts a capacity problem into timeouts, retries, and eventually a retry storm.

if revision_mismatch: fail readiness
if projected_cache > safe_capacity: reject before enqueue
if deadline_expires: cancel scheduler state and release blocks
if worker_draining: stop admission, finish bounded work, unload
if error_rate_spikes: route to the previous release fingerprint

Rollback must include the model artifact, tokenizer, template, runtime build, scheduler configuration, and API behavior. Rolling back weights alone can preserve the exact mismatch that caused the incident.

Exercise the states the happy-path benchmark avoids.

Kill one worker during prefill and another during decode. Slow a streaming client. Corrupt a readiness fingerprint. Exhaust the request queue without exhausting device memory. Fill cache blocks with long contexts. Cancel half the active requests. Delay one distributed rank. Each drill should have an expected reject, retry, drain, or rollback behavior.

A reliable drill proves:

bounded retries do not duplicate expensive work indefinitely;
deadlines and disconnects reclaim scheduler and cache state;
the router excludes cold, draining, and incompatible replicas;
alerts identify the boundary at fault rather than only reporting latency;
rollback restores the full serving fingerprint.

Run drills before and after scheduler or engine upgrades. Reliability is a property of the complete request path, so a faster kernel can still reduce reliability by changing memory headroom or failure timing.