Batch token work, not merely requests.
Static batching waits for a group, pads it to compatible shapes, and holds every slot until the slowest sequence finishes. Dynamic batching collects requests for a short boundary-time window. Continuous—or in-flight—batching rebuilds the active batch at each generation iteration, removes completed sequences immediately, and inserts newly admitted work into freed capacity.
Before execution
Simple, but padding and tail waste are high
Short queue window
More throughput at the cost of deliberate queue delay
Every token iteration
High utilization with a stateful scheduler
Token-budget slices
Less decode interference, more scheduling transitions
A useful scheduler tracks a token budget and a cache-block budget. It must decide how much prefill and decode work to mix, whether a long prompt may monopolize an iteration, when to preempt a sequence, and how to prevent a high-volume tenant from starving others. First-come-first-served is predictable but can suffer head-of-line blocking; priority classes require quotas and aging so low-priority work eventually progresses.
while running:
release_finished_and_cancelled()
budget = step_token_budget
batch = select_decode_sequences(budget, fairness_policy)
budget -= decode_tokens(batch)
batch += select_prefill_chunks(budget, free_cache_blocks)
logits = execute(batch)
sample_and_stream(logits)Preemption is not free. Swapping cache blocks to host memory spends bandwidth; recomputing a prefix spends model time. Log why each sequence was preempted and whether it was swapped, recomputed, rejected, or timed out. Scheduler policy belongs in the service version because it changes user-visible latency even when weights are unchanged.
The scheduler is a user-experience policy encoded in token budgets.
At each iteration the scheduler chooses which sequences advance. Favoring the largest possible batch can maximize throughput while making an interactive request wait behind long offline jobs. Strict first-come-first-served is simple but allows a single huge prefill to block every active decoder. Priority queues help, but without quotas and aging they can starve low-priority tenants indefinitely.
release finished, timed-out, and cancelled sequences
reserve decode slots for latency-sensitive work
allocate the remaining token budget to prefill chunks
apply tenant quotas and age waiting requests
execute one iteration and stream completed tokens
record every defer, preempt, reject, and cache releaseOne very long prompt plus short chats
Do short requests still receive a first token?
Two tenants with unequal arrival rates
Can the quieter tenant make progress?
Clients disconnect during decode
Are compute and cache reclaimed immediately?
Arrival rate exceeds service rate briefly
Does queue time recover after the burst?
Sustained excess token work
Are rejects bounded and intentional?
Preemption needs an explicit cost model. Swapping cache state to host memory consumes link bandwidth and may create a second bottleneck. Recomputing a prefix uses accelerator time but avoids transfer state. Dropping the request is cheapest for the server and most expensive for the user. Record which choice was made and why.
The service version should therefore include scheduler policy, token budget, maximum active sequences, prefill chunk size, priority classes, quota rules, and preemption mode. Two servers with the same weights but different schedulers deliver observably different products.
Replay policy decisions before buying more hardware.
A lightweight discrete-event simulator can consume recorded arrivals, prompt lengths, output lengths, priorities, and cancellation times. It need not model kernels perfectly; it only needs measured prefill and decode service curves. Compare token budgets, chunk sizes, quotas, and preemption policies on the same trace before testing the best candidates on hardware.
Inspect per-tenant wait time, starvation duration, completed tokens, preemptions, rejected work, and cache occupancy. Throughput can remain constant while one tenant absorbs nearly every slot. Fairness must therefore be measured per class rather than inferred from the global average.
Finally replay a burst followed by quiet traffic. A stable scheduler drains the queue and returns latency to baseline. If tail latency remains elevated, work is leaking, retries are amplifying load, or the policy has no path back from overload.