Post-training a language model.

Define the behavior before choosing the optimizer.

Post-training begins with a behavior specification: which instructions should be followed, which constraints outrank others, when the assistant should ask a question, which actions require permission, what counts as task completion, and which failures are unacceptable. A dataset is useful only when it encodes that contract consistently.

A base model, an assistant, and an agent are different artifacts. Pre-training learns broad statistical capability from next-token targets. Supervised tuning teaches the conversation role and response format. Preference learning and reinforcement learning shift the policy toward judgments or outcomes that ordinary text does not supply densely.

Keep those stages distinguishable in experiments. If the chat template, policy data, generation settings, and weights all change together, an observed improvement cannot be assigned to one cause. Preserve the base checkpoint, version every derivative policy, and test both newly learned behavior and retained base capability.

Use one canonical conversation schema across demonstration, preference, and evaluation data. Preserve role boundaries as explicit tokens or structured fields. Keep tool calls and tool results distinct from natural-language messages, and attach source, policy, annotator, and version metadata to every example.

Split by task and semantic cluster, not by individual response. If two variants of the same prompt land in train and evaluation sets, the score measures memorization of the annotation pattern. Balance task domains, difficulty, languages, refusal cases, tool-use cases, and response lengths explicitly.

Teach the response format with assistant-only loss.

Supervised fine-tuning starts from a pre-trained checkpoint and optimizes high-quality demonstrations. A conversation is serialized into one token sequence, but the loss mask normally selects only tokens the assistant is responsible for generating. System and user tokens provide conditioning; assistant tokens provide targets.

input_ids = serialize(messages)
labels = input_ids.clone()
labels[role != "assistant"] = -100
labels[padding] = -100
loss = model(input_ids, labels=labels).loss

Masking is easy to get subtly wrong at role separators, tool results, and multi-turn boundaries. Unit tests should decode every supervised span and verify that the first predicted assistant token sees the complete prompt but no future response token.

SFT data quality dominates raw volume. Strong demonstrations should be correct, direct, appropriately detailed, and internally consistent about policy. Oversampling one response style can make the model verbose or formulaic. Monitor task loss separately from format compliance, refusal precision, answer length, and held-out base capabilities.

Collect comparisons that isolate meaningful differences.

A preference record contains a prompt x, a preferred response y⁺, a rejected response y⁻, and the criteria behind the choice. The pair is informative only when the evaluator can make a stable distinction. Two equally good answers create noise; one obviously broken answer teaches little about the decision boundary.

Randomize left-right order and hide model identity. Control for length: evaluators often reward longer answers even when detail adds no value. Preserve per-criterion judgments before producing an overall preference so later audits can separate factual correctness from style or safety.

Mine pairs near the current policy’s capability frontier. If candidates are drawn only from an old or much weaker policy, the optimized model quickly outruns the dataset. Refreshing candidate generation and keeping an untouched evaluation pool prevents the signal from going stale.

Turn pairwise judgment into a learned scalar carefully.

A reward model reads the prompt and response, then emits a scalar. With the Bradley–Terry formulation, the probability that y⁺ is preferred is a sigmoid of the reward difference. Training minimizes the negative log-probability of the observed ordering.

Hold out prompts—not only pairs—to test generalization. Slice performance by language, response length, safety category, task difficulty, and candidate source. A high overall ranking accuracy can hide a model that rewards verbosity, familiar formatting, or confident tone.

The reward model is a proxy. Once a policy is optimized against it, small proxy errors become attractive strategies. Maintain adversarial sets, disagreement reviews, and periodic human comparison of high-reward outputs. The closer optimization pushes to the edge of the reward model’s training distribution, the less its scalar should be trusted without direct evidence.

Let the model practice where outcomes can be checked.

Supervised demonstrations ask the model to imitate solutions written by people or stronger systems. Reinforcement learning can instead let the policy sample many candidate trajectories and reinforce the ones that satisfy an external verifier. Mathematics, code, games, and constrained tool tasks are useful because correctness can often be checked without asking a judge to estimate quality from style.

The AlphaGo analogy in the lecture is structural: expert demonstrations establish a competent starting policy, then self-play creates experience beyond the human dataset because the game supplies an exact outcome. Language tasks rarely offer such a complete simulator, but the same pattern applies wherever tests, equations, schemas, or environment state can determine success.

A verifier changes the data bottleneck. The model can generate its own practice attempts, but only if the task distribution is diverse, the checker is difficult to exploit, and failed attempts are retained for analysis. A reward of one for passing a weak test invites reward hacking; require hidden tests, adversarial cases, resource limits, and independent evaluation.

Improve reward without letting the policy drift freely.

In PPO-style human-feedback training, the current policy samples a response. The reward model scores it, and a penalty compares the policy against a frozen reference model. A value head estimates expected return so the optimizer can compute lower-variance advantages.

The practical loop is rollout generation, reward computation, advantage estimation, several minibatch updates, and evaluation. Log reward-model score, KL, response length, entropy, value loss, clip fraction, task success, and independent human preference. A reward increase with collapsing entropy, exploding length, or falling task success is not progress.

Stability depends on rollout freshness, learning rate, KL control, batch size, reward normalization, value accuracy, and the number of optimizer passes over each rollout. Reuse a batch too aggressively and the policy becomes far from the policy that generated it. Move too conservatively and the run spends substantial compute for little behavior change.

Optimize pairwise preferences without a separate reward model.

Direct preference optimization compares how the current policy and a frozen reference policy score the chosen and rejected responses. It increases the relative log-probability of the chosen response while retaining an implicit constraint to the reference.

DPO removes rollout generation, value estimation, and a separately trained reward network from the optimization loop. That simplicity does not remove data risk. Pair quality, policy coverage, sequence-length treatment, reference choice, and β still determine the behavior learned.

chosen_margin  = logp_policy_chosen  - logp_ref_chosen
rejected_margin = logp_policy_rejected - logp_ref_rejected
loss = -logsigmoid(beta * (chosen_margin - rejected_margin)).mean()

Track chosen and rejected log-probabilities separately. A falling loss can come from increasing the chosen response, suppressing the rejected response, or both. Evaluate base capabilities and calibration after every checkpoint; preference optimization can narrow the policy if the pair distribution is too small or stylistically uniform.

Train the process when the outcome depends on a process.

For multi-step tasks, the supervision target is not only the final answer. The model must learn when to decompose, when to inspect the environment, which tool to call, how to read the observation, and when evidence is sufficient to stop. Store concise decision summaries rather than private internal monologues.

An autoregressive model receives a roughly fixed amount of computation per generated token. Producing intermediate tokens gives it more sequential compute and a writable scratch space in context. This helps only when training rewards useful decomposition and verification; extra length by itself can produce confident detours.

Tool calls should be trained as structured outputs with strict schemas. Tool results belong to a separate role and must be treated as untrusted data. Reward the verified task outcome, then keep process criteria distinct: required tool use, prohibited actions, total turns, retries, latency, and irreversible side effects.

Model behavior is only one layer of the security boundary. Sandboxes, allowlists, scoped credentials, user confirmation, output validation, rate limits, and audit logs must constrain what generated text can cause. Training should expose attempted overrides and malicious observations, while the runtime must remain safe even when the model makes the wrong decision.

Reasoning data needs its own quality controls. Long traces can contain unsupported steps, copied errors, or unnecessary verbosity. Outcome-based filtering alone keeps lucky reasoning. Add step-level checks where possible, compare alternate approaches, and include recovery examples in which an observation invalidates the current plan.

A post-trained agent should learn when to act, when to verify, and when to stop.

An assistant must know which information is in weights, context, or tools.

Pre-training compresses patterns into weights. The conversation supplies temporary working memory. Tools can provide current or exact external state. Post-training should teach the policy to distinguish these sources: answer stable, well-supported knowledge directly; use a tool when freshness or precision matters; and express uncertainty when neither source is adequate.

Hallucination is encouraged when the training format always contains an answer, even when the prompt asks about a nonexistent fact or a future event. Include examples that verify premises, decline unsupported specificity, ask for missing context, or call an appropriate tool. Tool use should be rewarded for correctness and necessity, not merely frequency.

Models also have imperfect self-knowledge. A training cutoff or system identity supplied in a prompt is context, not a fact the base weights can reliably derive about themselves. Keep version, date, permissions, available tools, and policy constraints in explicit system metadata.

Capability remains jagged. Tokenization makes spelling and character operations unlike ordinary human reading; stochastic generation makes repeated attempts differ; strong performance in one domain does not imply reliable performance in an adjacent one. Post-training should calibrate behavior around this uneven map rather than hide it behind a single confidence style.