Series 03 / Chapter 08: Speculative Decoding and Latency Levers

MODEL SERVINGCHAPTER 08 / 10

Optimize the phase that dominates the workload.

Fused attention kernels reduce intermediate memory traffic and avoid materializing the complete attention matrix. Fused normalization, activation, projection, and sampling kernels reduce launches and round trips to device memory. CUDA graphs or similar execution capture can remove repeated host-side launch overhead when shapes and control flow permit it.

Grouped-query or multi-query attention shares key/value heads across more query heads, shrinking the KV cache and its bandwidth. Prefix caching skips repeated prefill for exact shared prefixes. Both affect model or request contracts and must be validated under the real template and length distribution.

Speculative decoding uses a cheaper draft process to propose several tokens, then asks the target model to verify them together. Accepted tokens advance the sequence; at the first rejection, normal target sampling restores the correct target distribution. The gain depends on draft cost, acceptance rate, verification shape, and available accelerator headroom.

Useful speculative workgain ∝ accepted target tokens / (draft cost + verification cost)

A low acceptance rate adds work; a saturated large batch may have little spare capacity for the speculative path.

OptimizationLikely targetCommon trap

Fused kernels

Prefill and decode execution

Unsupported shapes or silent fallback

Prefix cache

Repeated system or document prefixes

Low exact-match rate or unsafe sharing

Speculation

Low-batch decode latency

Draft overhead exceeds accepted work

Long-context chunking

Decode smoothness during prefill

Too many tiny chunks and transitions

Change one lever at a time and retain output-parity checks. The correct optimization is workload-specific: an interactive coding assistant, an offline synthetic-data job, and a high-concurrency short-answer API should not share one unquestioned configuration.

Choose the runtime before tuning its cleverest feature.

Local CPU and consumer-device deployments often value portable quantized files, small memory footprints, and a simple server. High-concurrency GPU services value paged cache allocation, continuous batching, adapter support, and distributed execution. Agent workloads may add constrained decoding, tool-call parsing, prefix reuse, and long-lived streaming connections. These are different optimization problems even when they serve the same checkpoint.

WorkloadRuntime capability to prioritize

single-user local

Portable quantization, CPU/GPU offload, predictable memory, simple endpoint

interactive shared API

Tail latency, continuous batching, paged cache, cancellation

offline generation

Aggregate token throughput, large batches, fault recovery

tool-using agents

Schema-constrained output, stable templates, streaming tool events

many fine-tunes

Adapter multiplexing, revision-aware routing, cache isolation

Speculative decoding adds a draft path that proposes tokens and a target path that verifies them. The target distribution remains authoritative, but speed depends on how many proposals are accepted before a rejection. A draft model that is too slow, poorly aligned, or competing for already saturated compute can make the system slower.

Speculation break-evenaccepted_tokens x normal_decode_cost > draft_cost + verification_cost

Measure acceptance length and cost by prompt domain, not only as one average.

Prefix caching has a similarly conditional payoff. It helps only when rendered prefixes match exactly and remain safe to share. Template changes, dynamic timestamps, tenant-specific secrets, or different tool schemas can destroy the hit rate or create isolation risk.

Run an ablation table: baseline engine, one optimization enabled, semantic parity result, TTFT, inter-token latency, throughput, memory, and cost. The fastest credible configuration is the one that survives the same output and workload tests, not the one with the most flags enabled.

Make every speed claim survive an ablation.

Start from a stable baseline and enable one change at a time: fused attention, graph capture, prefix caching, speculative decoding, lower-precision cache, or a new batching policy. For each row, record semantic parity, TTFT, inter-token latency, throughput, memory, compilation time, and the workload slice that improved.

For speculation, report proposal length, acceptance length distribution, draft time, verification time, and rejected work. For prefix caching, report eligible requests, exact hits, bytes reused, invalidations, and cross-tenant isolation. These measurements explain why an optimization succeeds or fails instead of reducing the result to a single token-rate number.

After individual ablations, test combinations. Two optimizations can compete for memory or alter shapes enough to disable each other's kernels.