Evals and RL environments are the same machine with different jobs.

Start with three components.

Every useful task needs a prompt, an environment, and a grader. The prompt defines the objective. The environment gives the model a state to observe and actions it can take. The grader converts the resulting behavior into a score.

This is true whether the task asks an agent to repair software, operate a terminal, navigate a simulated computer, or complete a multi-step workflow. Without the environment, we only measure language. Without the grader, we have no stable signal. Without a well-formed prompt, we do not know which capability the score represents.

The architecture is shared. What changes is the destination of the score.

In an evaluation, the score returns to a researcher. It helps compare model checkpoints, choose training directions, or decide whether a change created a regression. In an RL environment, the score enters the optimization loop and contributes directly to changing model behavior.

An LLM environment is more than a prompt and a reward function.

The classical environment loop is compact: reset the world, return an observation, accept an action, transition the state, emit a reward, and say whether the episode is done. For language-model agents, each of those words hides a system boundary. An action may be a token sequence, a structured tool call, a shell command, or a browser operation. An observation may contain command output, files, screenshots, error messages, or a compact state summary.

A useful specification therefore names every component that can change the trajectory. The task identifies the capability under pressure. The prompt template decides what the policy sees. The initial state creates the starting world. Tools and their schemas define the action space. The execution backend determines what actions actually do. Observation rules decide which consequences become visible. The reward or rubric scores the result. The done rule ends the episode. Episode control handles resets, timeouts, seeds, and repeated rollouts. Transport carries all of this between the trainer, model, sandbox, and grader.

This larger contract explains why an RL dataset is not interchangeable with a supervised dataset. A row of prompt and answer text cannot reproduce the state transitions, tool failures, partial observations, resource costs, or alternative strategies that produced an outcome. The environment is the generator of experience, not merely a wrapper around examples.

A trajectory is the evidence trail of one attempt.

An episode begins with reset(). The environment creates an isolated state and returns the first observation. The policy chooses an action. The environment validates and executes it, changes the state, and returns the next observation, reward information, and termination status. This repeats until success, failure, timeout, cancellation, or a step budget ends the episode.

observation = env.reset(task=task, seed=seed)
while not observation.done:
    action = policy(observation, history)
    observation = env.step(action)
    trajectory.append({
        "state_ref": observation.state_ref,
        "action": action,
        "result": observation.result,
        "reward_parts": observation.reward_parts,
        "cost": observation.cost,
    })

The trace must distinguish what the model knew before an action from what became available afterward. Otherwise analysis can accidentally credit the policy with information revealed by the tool result or hidden grader. Record tool arguments, stdout and stderr, exit status, changed files, network requests, timing, resource usage, and the exact rubric version. For long episodes, store large artifacts by content hash and keep references in the trajectory.

Running several rollouts for the same task is especially valuable for stochastic policies. Two attempts can start from the same state and receive different rewards because one explores a useful action while another takes a shortcut. Training can use that contrast, but only if reset isolation ensures the second rollout cannot see files, caches, or answers left by the first.

Different environment types teach different kinds of competence.

A one-shot verifiable environment asks for one response and scores it with an exact or executable check. Mathematics, logic, and contained code problems fit this pattern when the answer can be verified reliably. These environments are scalable and easy to reset, but they teach less about information gathering, recovery, and long-horizon tool use.

A multi-turn tool environment exposes actions and observations over time. The policy must choose what to inspect, decide when evidence is sufficient, handle tool errors, preserve progress, and stop. Coding sandboxes, browser tasks, and scientific workflows belong here. The reward may still be verifiable, but the path matters because the same final state can be reached through safe work, destructive shortcuts, or leaked information.

Do not collapse these into one benchmark average. A model that improves on exact-answer tasks may not improve at recovery after a failed command. A model that excels in a permissive sandbox may fail when tools enforce schemas and least privilege. Evaluation slices should preserve the environment family so the learning signal remains interpretable.

The user of the signal changes the data you need.

A human researcher is relatively sample-efficient. They can look at a smaller set of carefully constructed failures, recognize a pattern, and choose the next experiment. Their time is scarce, so noisy or misleading cases are expensive. An evaluation suite should therefore emphasize precision, stability, and diagnostic value.

An optimization algorithm has the opposite appetite. It needs many more experiences. Reusing a narrow set of tasks creates brittle learning, so an RL environment needs breadth, diversity, and enough variation to support generalization.

The distinction is not “quality versus quantity.” Both require quality. The distinction is what quality means for the loop: low-noise evidence for a human decision, or diverse experiences for robust learning.

A useful task sits at the edge of capability.

If every model receives a perfect score, the task cannot tell us which system is better. If every model receives zero, the result is equally unhelpful. The task becomes decision-relevant only when current systems display meaningful variation.

This is why good task design begins with a real failure. Try to use the model. Find where it stops being dependable. Then isolate enough of that situation to reproduce the difficulty without removing the thing that made it difficult.

The target moves. A task that exposed a capability gap last year may be saturated today. That is not a failure of evaluation; it is evidence that the measurement did its job and a sharper question is now required.

The grader does not merely observe behavior. It shapes it.

A simple software test might confirm that a happy path works while missing the interactions a real user will try. Once a model is optimized against that test, passing it can become easier than building a genuinely reliable feature.

This is the central danger of a weak proxy. The score rises, but the underlying capability does not improve in the way users care about. More data does not remove a systematic bias in the grader; it can reinforce it.

The right question is not only “Does the grader work?” It is “What model does this grader create after thousands of optimization steps?”

Reward hacking is an environment failure before it is a model surprise.

An optimizer searches for behavior that receives reward. It does not know which causal story the designer intended. If a coding reward pays for a program that runs without error, a policy may learn to catch every exception and print a constant instead of solving the task. The scalar reward can rise smoothly while actual correctness remains flat.

This is why aggregate learning curves are insufficient. Read trajectories throughout training. Compare high-reward traces with independent outcome checks. Run trivial baselines, no-op policies, random actions, and deliberately adversarial solutions before giving the environment to an optimizer. If a weak baseline earns meaningful reward, the proxy is already leaking signal.

Security boundaries are part of measurement validity. An agent that can reach the answer key, modify the tests, or escape the sandbox is not demonstrating the target capability. It is demonstrating that the environment enforced the wrong world. Restricting egress and isolating the grader often lowers headline reward while making the signal more trustworthy.

The practical rule is simple: whenever reward improves unusually quickly, inspect what behavior became cheaper. The most useful debugging artifact is not the curve; it is the high-scoring trajectory beside an independent account of what actually changed in the world.

Build the learning loop deliberately.

When I think about an RL environment, I start from the capability gap and work backward. The virtual world, available tools, task instructions, and scoring logic should all preserve the difficulty I want the model to learn from.

1. 01
   Begin with a real pain point.

   A concrete failure is more informative than a capability imagined in the abstract.

2. 02
   Preserve the source of difficulty.

   Do not simplify the scenario until the very behavior you wanted to measure disappears.

3. 03
   Make success verifiable.

   Prefer observable state changes and outcomes over confident-looking language.

4. 04
   Audit the grader adversarially.

   Search for cheap ways to score well without developing the intended capability.

5. 05
   Design for the next model.

   The task should remain useful as capabilities improve, not only separate today’s systems.