Series 03 / Chapter 06: Parallel Inference Across Accelerators

MODEL SERVINGCHAPTER 06 / 10

Choose the communication pattern that matches the bottleneck.

If one accelerator holds the checkpoint, replicas provide the simplest scale-out: each replica serves independent requests and the router balances load. When the model does not fit or one replica misses the latency target, the model itself must be partitioned.

StrategyPartitionPrimary cost

Replica / data parallel

Independent full models

Duplicate weights and uneven routing

Tensor parallel

Matrix dimensions within each layer

Collective communication every layer

Pipeline parallel

Consecutive layer stages

Stage bubbles and activation transfers

Expert parallel

Sparse experts across devices

Token all-to-all and routing imbalance

Prefill/decode split

Execution phase

KV transfer and cross-pool routing

Tensor parallelism reduces per-device weight memory and can lower single-request latency when devices have a fast interconnect, but collective operations occur throughout the network. Pipeline parallelism places different layers on different stages; utilization depends on enough concurrent microbatches to fill the pipeline. Expert parallelism is natural for sparse models but makes router balance and all-to-all traffic part of the latency path.

Benchmark the whole topology. A kernel speedup can disappear behind communication, and adding devices can increase latency when the batch is too small to amortize collectives. Record topology, link type, shard count, collective time, and per-rank memory beside throughput.

Add devices only when the communication bill is understood.

The first question is whether one accelerator can hold the weights, runtime workspace, and target cache capacity. If it can, independent replicas are usually the cleanest scale-out unit: they avoid per-layer collectives and isolate failures. Partition the model only when memory fit or single-request latency requires it.

DecisionEvidence required

replicate

One device fits the service contract; routing keeps replicas balanced.

tensor parallel

Fast links make layer-wise collectives cheaper than the latency saved.

pipeline parallel

Enough concurrent microbatches exist to fill stages without large bubbles.

expert parallel

Router balance and all-to-all traffic remain stable under real token mixes.

prefill/decode split

Phase specialization repays KV-transfer and cross-pool routing cost.

Topology changes the failure domain. A request spread across four tensor-parallel ranks fails if any rank fails. A replica pool can lose one worker while other replicas continue. Pipeline stages require coordinated draining. Expert-parallel systems must also handle skew: a popular expert can saturate its rank while aggregate accelerator utilization looks healthy.

Useful scale-outspeedup = single_device_time / distributed_time

Distributed time includes collectives, synchronization, routing, padding, and imbalance.

Benchmark at the batch sizes the product will actually use. A topology that wins at a large offline batch may lose for a single interactive sequence because collective latency cannot be amortized. Record per-rank weight memory, cache memory, compute time, collective time, link utilization, and straggler delay.

Finally, make the routing layer revision-aware. A replica must advertise the exact model, tokenizer, adapter set, numerical engine, and cache format it serves. Sending a request to a merely "healthy" but incompatible worker is a semantic failure, not a load-balancing success.

Compare topologies at equal service quality.

Hold the model revision, numerical format, workload, output parity threshold, and total accelerator count fixed. Measure a replica baseline before partitioning. For each topology, report first-token latency, decode latency, throughput, per-rank memory, collective time, and failure behavior. Include both one-request and saturated cases.

A topology that increases peak throughput but misses the interactive latency target is not a drop-in improvement. A topology that fits longer contexts by reducing per-rank weight memory may still lose capacity to communication buffers. A topology that depends on all ranks should be tested with one rank delayed or removed.

Communication sharecollective_time / end_to_end_step_time

Track the distribution by layer and batch shape; a single mean hides stragglers.