Model serving and inference.

Start with a service-level objective, not a GPU.

Offline generation optimizes the time to finish a bounded batch. An interactive service must satisfy a distribution of requests arriving over time. The objective is therefore multi-dimensional: make the first token arrive quickly, keep later tokens flowing smoothly, sustain enough concurrent work, stay within a memory and cost budget, and reject overload predictably.

State percentiles and workload conditions with every number. A mean hides queue spikes; a tokens-per-second record without input lengths, output lengths, concurrency, and hardware cannot be reproduced. Separate client-observed latency from server execution so network and application overhead are not attributed to the model engine.

Define semantic requirements beside performance: tokenizer and chat-template version, maximum context, stop conditions, sampling policy, deterministic mode, tool-call schema, log-probability behavior, cancellation semantics, and compatibility rules. A faster server that silently changes tokenization or sampling is a different system.

Make every boundary explicit and observable.

A production request passes through more than a model call. The edge authenticates the caller and applies rate limits. The API validates the prompt, generation parameters, and context budget. A tokenizer converts messages to IDs. Admission control estimates the work and memory. A router selects a compatible replica. The scheduler forms token batches. Model workers produce logits, the sampler chooses tokens, and a streaming layer converts IDs back into text.

Streaming adds backpressure. If a client reads slowly or disconnects, the server should release its sequence and cache blocks quickly. Cancellation must propagate from the edge to the scheduler and workers; otherwise invisible abandoned requests consume the scarce decode budget.

One request contains two very different workloads.

During prefill, the engine processes the prompt tokens and writes their keys and values into the cache. All prompt positions can be evaluated in parallel under the causal mask, so large matrix operations usually use the accelerator efficiently. The cost grows sharply with prompt length because each query attends across its permitted prefix.

During decode, the engine generates one new token per active sequence per iteration. The new query attends over cached keys and values, then the chosen token extends the cache. The arithmetic per iteration is smaller, but model weights and cache state must be accessed repeatedly. Small decode batches can therefore be dominated by memory bandwidth, communication, and launch overhead.

Aggregated serving runs both phases on the same worker pool. A long prefill can delay decode steps and produce visible pauses for existing users. Chunked prefill limits how many prompt tokens enter one scheduling iteration, allowing decode work to interleave. Disaggregated serving places prefill and decode on different worker pools, which permits independent scaling but adds KV-cache transfer, routing state, and failure modes.

Measure the phases separately before selecting an architecture. Disaggregation is useful only when the workload imbalance and interference it removes are worth the extra network path and operational complexity.

Cache memory is the admission currency.

Every decoder layer stores a key and value vector for every live token. With grouped-query attention, the cache uses the number of key/value heads rather than the number of query heads. Before sharding and allocator metadata, the approximate bytes for one sequence are:

For 32 layers, 8 key/value heads, head dimension 128, and two-byte cache values, each token consumes about 128 KiB per sequence. An 8,192-token sequence is therefore about 1 GiB before allocator overhead and any device partitioning. Batch count alone is not a capacity measure: eight short prompts and eight near-limit prompts require radically different cache budgets.

Contiguous allocation reserves a large region for each possible sequence and wastes space when lengths vary. Paged allocation divides the cache into fixed-size blocks, maps logical token positions to physical blocks, and returns blocks as sequences finish. It reduces external fragmentation and lets the scheduler admit work using available blocks rather than worst-case sequence reservations.

Prefix caching can reuse blocks for an identical token prefix, reducing repeated prefill work. Cache identity must include every value that changes model state: model revision, adapter, tokenizer, chat template, positional policy, and relevant multimodal inputs. Tenant boundaries and access policy must prevent one caller from observing or timing private cached prefixes.

Batch token work, not merely requests.

Static batching waits for a group, pads it to compatible shapes, and holds every slot until the slowest sequence finishes. Dynamic batching collects requests for a short boundary-time window. Continuous—or in-flight—batching rebuilds the active batch at each generation iteration, removes completed sequences immediately, and inserts newly admitted work into freed capacity.

A useful scheduler tracks a token budget and a cache-block budget. It must decide how much prefill and decode work to mix, whether a long prompt may monopolize an iteration, when to preempt a sequence, and how to prevent a high-volume tenant from starving others. First-come-first-served is predictable but can suffer head-of-line blocking; priority classes require quotas and aging so low-priority work eventually progresses.

while running:
    release_finished_and_cancelled()
    budget = step_token_budget
    batch = select_decode_sequences(budget, fairness_policy)
    budget -= decode_tokens(batch)
    batch += select_prefill_chunks(budget, free_cache_blocks)
    logits = execute(batch)
    sample_and_stream(logits)

Preemption is not free. Swapping cache blocks to host memory spends bandwidth; recomputing a prefix spends model time. Log why each sequence was preempted and whether it was swapped, recomputed, rejected, or timed out. Scheduler policy belongs in the service version because it changes user-visible latency even when weights are unchanged.

Choose the communication pattern that matches the bottleneck.

If one accelerator holds the checkpoint, replicas provide the simplest scale-out: each replica serves independent requests and the router balances load. When the model does not fit or one replica misses the latency target, the model itself must be partitioned.

Tensor parallelism reduces per-device weight memory and can lower single-request latency when devices have a fast interconnect, but collective operations occur throughout the network. Pipeline parallelism places different layers on different stages; utilization depends on enough concurrent microbatches to fill the pipeline. Expert parallelism is natural for sparse models but makes router balance and all-to-all traffic part of the latency path.

Benchmark the whole topology. A kernel speedup can disappear behind communication, and adding devices can increase latency when the batch is too small to amortize collectives. Record topology, link type, shard count, collective time, and per-rank memory beside throughput.

Quantization changes memory traffic and the output distribution.

Reducing precision can make a model fit, allow more cache blocks, and reduce bandwidth per generated token. Weight-only quantization primarily reduces the cost of reading weights and is often attractive for decode. Weight-and-activation quantization can accelerate more operations but demands a supported kernel path and careful calibration. The KV cache can be stored at lower precision independently of the weights.

Memory reduction is not the same as latency reduction. Dequantization overhead, unsupported layers, small batch shapes, or frequent format conversions can erase the gain. Report the exact scheme, group size, scales, calibration set, fallback operations, and kernel implementation rather than a label such as “4-bit.”

Test parity at three levels: compare logits on fixed prefixes, compare deterministic outputs under fixed seeds and settings, then rerun behavioral evaluations. A small average logit error can still alter a near-tied token choice and cause a long generated sequence to diverge.

Optimize the phase that dominates the workload.

Fused attention kernels reduce intermediate memory traffic and avoid materializing the complete attention matrix. Fused normalization, activation, projection, and sampling kernels reduce launches and round trips to device memory. CUDA graphs or similar execution capture can remove repeated host-side launch overhead when shapes and control flow permit it.

Grouped-query or multi-query attention shares key/value heads across more query heads, shrinking the KV cache and its bandwidth. Prefix caching skips repeated prefill for exact shared prefixes. Both affect model or request contracts and must be validated under the real template and length distribution.

Speculative decoding uses a cheaper draft process to propose several tokens, then asks the target model to verify them together. Accepted tokens advance the sequence; at the first rejection, normal target sampling restores the correct target distribution. The gain depends on draft cost, acceptance rate, verification shape, and available accelerator headroom.

Change one lever at a time and retain output-parity checks. The correct optimization is workload-specific: an interactive coding assistant, an offline synthetic-data job, and a high-concurrency short-answer API should not share one unquestioned configuration.

Overload must be a designed state.

Capacity depends on the joint distribution of prompt length, output length, arrival rate, concurrency, cache reuse, and sampling settings. Request count is insufficient because one 100-token answer and one 10,000-token answer occupy the decode scheduler for very different durations. Maintain separate prompt-token and generated-token budgets, then estimate memory before admission.

Admission control should reject or defer work before the device reaches an unrecoverable memory state. Bound queue length, prompt tokens, generated tokens, live cache blocks, and per-tenant concurrency. Apply deadlines and propagate cancellation. Use backpressure rather than accepting unlimited work and converting every request into a timeout.

Autoscaling on accelerator utilization alone is late and ambiguous: a server can be memory-full with modest arithmetic use, or compute-busy while the queue is healthy. Combine queue delay, admitted token work, cache occupancy, active sequences, deadline misses, and load per replica. Keep warm capacity when model loading and compilation take longer than the service’s recovery objective.

Fail closed on contract mismatch. A replica with the wrong tokenizer, adapter, numerical engine, or model revision should not receive traffic merely because its health endpoint responds. Readiness should include a fixed prompt probe, expected token IDs, and a small deterministic output or logit checksum.