Series 03 / Chapter 01: Inference Begins With a Service Contract

MODEL SERVINGCHAPTER 01 / 10

Start with a service-level objective, not a GPU.

Offline generation optimizes the time to finish a bounded batch. An interactive service must satisfy a distribution of requests arriving over time. The objective is therefore multi-dimensional: make the first token arrive quickly, keep later tokens flowing smoothly, sustain enough concurrent work, stay within a memory and cost budget, and reject overload predictably.

MetricBoundaryWhat it exposes

Time to first token

Request accepted → first streamed token

Queueing, tokenization, prefill, and first decode step

Inter-token latency

One streamed token → the next

Decode scheduling and per-step execution

End-to-end latency

Request accepted → finish reason

Full user wait, including output length

Token throughput

Prompt and generated tokens per second

Accelerator utilization and batching

Request throughput

Completed requests per second

Capacity under the actual length mix

State percentiles and workload conditions with every number. A mean hides queue spikes; a tokens-per-second record without input lengths, output lengths, concurrency, and hardware cannot be reproduced. Separate client-observed latency from server execution so network and application overhead are not attributed to the model engine.

Define semantic requirements beside performance: tokenizer and chat-template version, maximum context, stop conditions, sampling policy, deterministic mode, tool-call schema, log-probability behavior, cancellation semantics, and compatibility rules. A faster server that silently changes tokenization or sampling is a different system.

The deployable artifact is larger than the weight file.

A checkpoint can load successfully and still behave incorrectly. The serving contract also includes the tokenizer revision, chat template, beginning- and end-of-sequence policy, generation defaults, adapter lineage, numerical format, context limit, and tool-call grammar. If any of these change between training and serving, the runtime may produce repetitions, malformed turns, premature stops, or outputs that look like a model regression.

ArtifactRelease question

weights + adapters

Are adapters merged, loaded dynamically, or selected per request?

tokenizer + template

Does the exact training-time conversation format survive export?

special tokens

Which IDs begin, separate, and terminate turns?

numerical format

Which tensors were quantized, calibrated, or left at higher precision?

runtime manifest

Which engine, kernels, flags, and hardware topology produced the result?

Engine selection follows the workload. A compact CPU or laptop deployment values a portable file format and low memory overhead. A shared accelerator service values continuous batching, paged cache management, adapter multiplexing, and high token throughput. Structured generation and tool-heavy agents may instead prioritize constrained decoding, parser compatibility, and stable streaming events. There is no universally fastest engine because the model shape, hardware, context distribution, and concurrency determine the bottleneck.

release_fingerprint = hash(
    model_revision,
    adapter_revision,
    tokenizer_files,
    chat_template,
    special_token_ids,
    quantization_config,
    engine_build,
    serving_flags,
)

Put that fingerprint in logs, readiness responses, benchmark reports, and every generated trajectory. It turns a vague statement such as "the server changed" into a reproducible comparison between two complete inference systems.

Run a release review before running a load test.

Place the reference and candidate manifests side by side. For every difference, state whether it is intentional and which test covers it. A new engine build needs semantic parity tests; a new template needs rendered-token fixtures; a new quantization needs task and logit comparisons; a new scheduler needs workload and fairness tests. If a field cannot be compared, the release is not yet reproducible.

01
Load from an empty machine.
Prove that the artifact contains every required file and does not depend on a developer cache.
02
Render fixed conversations.
Store expected text and token IDs for plain chat, tools, multiple turns, and empty system prompts.
03
Fail incompatible workers.
Readiness must reject the wrong revision rather than letting the router discover it through user traffic.