Series 03 / Chapter 02: Anatomy of an Inference Request

MODEL SERVINGCHAPTER 02 / 10

Make every boundary explicit and observable.

A production request passes through more than a model call. The edge authenticates the caller and applies rate limits. The API validates the prompt, generation parameters, and context budget. A tokenizer converts messages to IDs. Admission control estimates the work and memory. A router selects a compatible replica. The scheduler forms token batches. Model workers produce logits, the sampler chooses tokens, and a streaming layer converts IDs back into text.

authenticate→validate→tokenize→admit→schedule→execute→sample→stream

Trace fieldReason to preserve it

request_id, tenant_id

Join events while keeping quotas and cache ownership distinct

model_revision, tokenizer_revision

Reproduce the exact serving contract

prompt_tokens, max_new_tokens

Estimate work and explain latency

sampling_config, seed

Separate engine changes from decoding changes

queue_ms, prefill_ms, decode_ms

Locate time instead of reporting one opaque total

finish_reason, error_code

Distinguish normal stop, limit, cancellation, rejection, and failure

Streaming adds backpressure. If a client reads slowly or disconnects, the server should release its sequence and cache blocks quickly. Cancellation must propagate from the edge to the scheduler and workers; otherwise invisible abandoned requests consume the scarce decode budget.

Protocol compatibility is not semantic compatibility.

Many runtimes expose an OpenAI-compatible endpoint, which is useful because clients can share one transport. But matching a route name and JSON shape does not guarantee matching behavior. Streaming chunk order, usage accounting, stop reasons, tool-call deltas, log-probability fields, error codes, and cancellation semantics can still differ. Treat the API as a versioned contract and test the events a real client receives.

BoundaryRecordFailure exposed

Admission

deadline, tenant, prompt tokens, requested output

Work accepted without capacity

Template

rendered text and token IDs

Role or special-token drift

Scheduler

queue time, batch membership, preemption

Head-of-line blocking

Generation

seed, sampler, stop rule, finish reason

Irreproducible or truncated output

Stream

first-byte time, chunk times, disconnect

Client-visible stalls and wasted decode

A robust request envelope carries a request ID, model revision, input or messages, maximum output tokens, sampling configuration, tool schemas, deadline, and optional idempotency key. The server derives token counts and projected cache use before admission. Once generation begins, cancellation must propagate from the socket through the scheduler to the cache allocator; otherwise a disconnected client continues consuming tokens invisibly.

POST /v1/chat/completions
  validate schema and auth
  render the pinned chat template
  tokenize and estimate cache blocks
  admit, defer, or reject before execution
  stream typed events with a terminal finish reason
  release cache blocks on finish, timeout, or disconnect

Readiness should exercise this path rather than merely returning HTTP 200. A small fixed prompt can verify tokenizer IDs, template rendering, model revision, deterministic output, and streaming completion. That probe catches a healthy process serving the wrong artifact.

Test the stream as a sequence of events.

Build client tests that consume chunks exactly as production clients do. Include normal completion, stop-sequence completion, maximum-token completion, tool calls split across multiple chunks, malformed requests, authentication failure, deadline expiry, and disconnect during generation. Assert both the event sequence and cleanup behavior.

A request-path release should answer:

Is the first event emitted only after admission succeeds?
Does every successful stream end with one unambiguous finish reason?
Are usage counts based on the same tokenizer as admission?
Does cancellation remove scheduler state and cache blocks?
Are retryable and terminal errors distinguishable?

Log server timestamps at admission, enqueue, first scheduled step, first emitted token, final token, and cleanup. Client-observed timing can then be decomposed without guessing which boundary spent the latency.