← Model-serving series

Series 03 / Chapter 10

Load Testing and the Evidence Bundle

Benchmark realistic workloads across tail latency, throughput, cache pressure, failures, output quality, and cost.

Vikram Kharvi13 Aug 20265 min read

SERIES 03 / CHAPTER 10

10
MODEL INFERENCEBENCHMARK THE WORKLOAD

REQUEST → COMPUTE → TOKEN → EVIDENCE

  • Latency
  • Memory
  • Quality
MODEL SERVINGCHAPTER 10 / 10

Benchmark a workload, not a demo prompt.

Create a replayable workload manifest containing prompt-length and output-length distributions, arrival process, concurrency, sampling settings, prefix-reuse rate, cancellation rate, and tenant mix. Run warmup separately, then hold hardware, model revision, and request corpus fixed while changing one serving parameter.

LayerEvidenceFailure exposed
Semantic parity

Tokenizer IDs, fixed-prefix logits, deterministic outputs

Engine or precision changed behavior

Latency

p50/p95/p99 first-token, inter-token, and end-to-end

Queue spikes and decode stalls

Capacity

Request and token throughput at rising load

Saturation knee and unstable overload

Resources

Weight memory, cache occupancy, bandwidth, collectives

The actual hardware bottleneck

Reliability

Rejects, timeouts, cancellations, OOMs, restarts

Work lost or hidden during stress

Plot throughput against tail latency while increasing arrival rate. The useful operating point lies before the saturation knee where queue time starts rising rapidly. Repeat with long prompts, long outputs, bursty arrivals, mixed tenants, cache hits and misses, worker loss, slow clients, and cancellation storms. A configuration is production-ready only if degradation is bounded and observable.

ArtifactRequired contents
model_contract

Weights, tokenizer, template, adapters, numerical format, sampling defaults

engine_manifest

Runtime revision, kernels, build flags, parallel topology, cache format

scheduler_config

Token budget, batch limit, prefill policy, priorities, preemption

workload_manifest

Length distributions, arrival pattern, concurrency, seeds, corpus hash

evidence_report

Semantic parity, latency percentiles, throughput, resources, failures, cost

Performance is part of intelligence.

If a strong checkpoint cannot deliver useful tokens inside the user’s latency, availability, and cost envelope, its capability is inaccessible. Serving quality is therefore the joint result of model design, memory allocation, scheduling policy, numerical implementation, and operational discipline.

Primary papers and official implementation references

The papers establish the algorithms; the official runtime documentation supplies current implementation contracts. The architecture, checklists, examples, and production recommendations are Vikram Kharvi’s synthesis.

A serving benchmark should be difficult to game.

Begin with a frozen workload manifest: prompt and output-length distributions, arrival process, concurrency, prefix reuse, sampling, tools, cancellation, and tenant mix. Separate warmup from measurement. Run at increasing arrival rates until tail latency rises sharply, then choose an operating point with headroom before that saturation knee.

GateRequired evidence
artifact parity

Tokenizer IDs, chat template, special tokens, adapter lineage, fixed-prefix logits

protocol parity

Streaming events, finish reasons, usage, errors, cancellation, tool schemas

quality parity

Task slices, long generations, tool calls, multilingual and adversarial prompts

performance

p50/p95/p99 TTFT, inter-token and end-to-end latency, token throughput

capacity

Cache occupancy, saturation knee, queue recovery, reject behavior, cost

resilience

Worker loss, slow clients, bursts, cancellations, cold starts, rollback

Measure output length as well as token speed. A configuration that produces tokens twice as fast but systematically generates twice as many tokens does not halve user wait or cost. Likewise, throughput gained by relaxing stop behavior, dropping tool constraints, or changing the prompt template is not a serving improvement; it is a different product.

Publish the workload hash, artifact fingerprint, hardware topology, engine build, serving flags, numerical formats, and raw percentile distributions. One headline number cannot distinguish an implementation improvement from a friendlier workload.

Deployment context and implementation references

The linked guides inform the deployment workflow, export, runtime, and troubleshooting coverage. The system model, calculations, release gates, and recommendations here are an original synthesis.

Publish enough detail for someone else to reproduce the claim.

A serving report should begin with the user contract and end with raw evidence. State hardware, topology, model and adapter revisions, tokenizer and template hashes, numerical formats, engine build, scheduler settings, cache configuration, workload hash, warmup, measurement duration, and confidence intervals or repeated-run variation.

Present latency distributions, throughput-versus-latency curves, cache occupancy, failures, rejects, cancellations, output lengths, and cost. Include at least a few concrete traces from normal operation and overload. A dashboard screenshot without the workload and artifact fingerprints is not reproducible evidence.

The final decision should name the safe operating point and its headroom: maximum admitted token work, expected concurrency, saturation signal, reject policy, rollback trigger, and quality threshold. This turns a benchmark from a victory lap into an operational contract.