Series 03 / Chapter 03: Prefill and Decode Are Different Workloads

MODEL SERVINGCHAPTER 03 / 10

One request contains two very different workloads.

During prefill, the engine processes the prompt tokens and writes their keys and values into the cache. All prompt positions can be evaluated in parallel under the causal mask, so large matrix operations usually use the accelerator efficiently. The cost grows sharply with prompt length because each query attends across its permitted prefix.

During decode, the engine generates one new token per active sequence per iteration. The new query attends over cached keys and values, then the chosen token extends the cache. The arithmetic per iteration is smaller, but model weights and cache state must be accessed repeatedly. Small decode batches can therefore be dominated by memory bandwidth, communication, and launch overhead.

PhaseUnit of workDominant pressure

Prefill

Many prompt tokens per sequence

Compute, attention workspace, long-prompt queueing

Decode

One new token per active sequence

Weight/cache bandwidth and step latency

KV transfer

Move prompt state to decode workers

Network bandwidth and handoff delay

Aggregated serving runs both phases on the same worker pool. A long prefill can delay decode steps and produce visible pauses for existing users. Chunked prefill limits how many prompt tokens enter one scheduling iteration, allowing decode work to interleave. Disaggregated serving places prefill and decode on different worker pools, which permits independent scaling but adds KV-cache transfer, routing state, and failure modes.

Measure the phases separately before selecting an architecture. Disaggregation is useful only when the workload imbalance and interference it removes are worth the extra network path and operational complexity.

A latency budget must name the phase that spends it.

Consider an interactive request with a 4,096-token prompt and a 256-token answer. Prefill processes thousands of prompt positions in parallel and largely determines time to first token. Decode then performs 256 sequential iterations and largely determines inter-token latency and total completion time. Reporting one blended tokens-per-second number conceals both user experiences.

First-token budgetTTFT = queue + tokenize + prefill + first_decode + network

The dominant term changes with prompt length, batch composition, and queue pressure.

Completion budgetlatency = TTFT + sum(inter_token_latency_2...N)

Output length multiplies small decode delays into a large user-visible wait.

Chunked prefill divides long prompts into token-budgeted pieces so short decode steps can continue between chunks. This improves responsiveness under mixed traffic, but too-small chunks add scheduler transitions and kernel inefficiency. The tuning variable is therefore not simply "enable chunking"; it is the chunk size and prefill-to-decode policy under the actual prompt distribution.

ExperimentHold constant

prefill curve

Batch prompts by length and plot TTFT against prompt tokens.

decode curve

Fix prompt state and plot inter-token latency against active sequences.

mixed load

Combine long prefills and active decodes; inspect tail stalls.

phase split

If using separate prefill and decode workers, include KV-transfer time.

Disaggregating the phases can let each worker pool use a different batching policy, but it introduces cache transfer, routing, and failure coordination. It is worthwhile only when the separation reduces more contention than the transfer path adds. Measure the boundary rather than assuming architectural elegance becomes speed.

Measure a grid, not one convenient prompt.

Cross prompt lengths such as 128, 1K, 4K, and 16K tokens with output lengths such as 32, 256, and 1K. Repeat at several concurrency levels. Report TTFT, inter-token latency, end-to-end latency, and both prompt and generated-token throughput. This exposes whether an optimization helps prefill, decode, or only a narrow batch shape.

Then introduce one long prefill while short decodes are active. Compare no chunking, several chunk sizes, and any phase-disaggregated design. Inspect p95 and p99 token gaps, not only mean throughput. The winning policy is the one that protects interactive progress while keeping enough prefill work on the device.

Every inference benchmark is a statement about a workload distribution.