A trajectory should be a versioned, ordered record with six required blocks. It needs a run envelope, the task, the environment, a turn-by-turn event stream, the outcome, and the measurements used to judge the run.
Start with one explicit trajectory schema.
A trajectory should not be assembled differently by every experiment. Use a stable schema so runs can be replayed, compared, filtered, and scored without guessing what a field means. At minimum, the record should consist of these six blocks:
schema_version, trajectory_id, agent and prompt versions, start and end times, status, and termination reason.
The original instruction, constraints, acceptance conditions, and maximum turn budget.
Initial state, available and prohibited tools, permissions, dependency versions, files, applications, and content hashes.
Ordered turn IDs, timestamps, messages, subgoals, tool calls, observations, state changes, errors, and artifact references.
Final response, terminal state, produced artifacts, unresolved errors, and the reason execution stopped.
Total turns, counts by tool, latency, tokens, cost, retries, criterion results, weights, scores, and evaluator errors.
Every event in turns[] should carry a stable ID and link its action to the resulting observation, state delta, and artifacts. Large outputs can live outside the record, but the trajectory must retain their location, content hash, summary, and any truncation marker.
Pair every action with its observation and state change.
A good event stream preserves order and causality. For every tool call, record its name, structured arguments, start time, duration, result, and error status. Then record what changed: a file was written, a page navigated, a test failed, or the environment returned a new observation.
Actions without observations hide whether the agent noticed consequences. Observations without action identifiers make it difficult to attribute a failure. Stable step IDs, parent-child links, and artifact references turn a long sequence into something that can be replayed and analyzed.
The unit of a trajectory is not a message. It is action, consequence, and updated state.
Preserve raw tool outputs when they are small and durable. For large outputs, store a reference, content hash, summary, and truncation marker. Silent truncation is dangerous because it makes the historical record look complete when it is not.
Capture intent without demanding a private monologue.
Reviewers need enough decision context to distinguish deliberate strategy from accidental success. That does not require storing hidden internal reasoning. A concise decision record is safer and more useful: the current subgoal, the reason for the next action, the evidence being used, and the uncertainty that might trigger clarification or escalation.
Record plan revisions, abandoned approaches, recovery attempts, and moments when the agent chose not to act. These are often the most informative steps. A trajectory that preserves only successful actions removes the evidence needed to study resilience.
- 01Current objective.
What local result is this step trying to produce?
- 02Decision summary.
Which observation justifies the selected action?
- 03Uncertainty.
What is unknown, assumed, or awaiting confirmation?
- 04Recovery.
How did the plan change after an error or surprising result?
Store the checks that prove both outcome and process.
An agent saying that the task is complete is not evidence. Capture the final artifacts, the terminal environment state, the reason execution stopped, and the checks that support success. Outcome checks can verify file existence and contents, command exit status and output, values in structured data, HTTP responses, or image dimensions and similarity.
Process checks belong in the trajectory too. Store counts by tool so a reviewer can verify that a required tool was used at least a minimum number of times, or that a prohibited tool was never used. Store the total turn count beside the declared maximum so efficiency penalties can be computed from the record rather than inferred later.
Keep these dimensions separate before combining them. A single reward hides whether a result was correct but inefficient, polished but incomplete, or successful only because a critical check never ran. Store every criterion's type, weight, optional status, isolation mode, raw result, final score, reviewer rationale, and evaluation error.
Artifacts and final state
What changed?
Executable checks
Does it work?
Criterion-level judgment
Is the work good?
Tool counts and turns
How was it achieved?
Efficiency and safety belong in the record.
Two trajectories can reach the same answer with very different operational value. Track turn count, tool count, repeated actions, latency, token usage, cost, retries, and time spent in each stage. These measurements expose loops, unnecessary exploration, and fragile dependence on one expensive step.
Capture safety boundaries too: permission requests, denied actions, secret redactions, data classifications, and external side effects. Sensitive values should be replaced with stable redaction tokens so the sequence remains understandable without retaining credentials or personal data.
Evaluation steps that execute commands or modify files should run independently where possible. Isolation prevents one check from changing the workspace seen by another and makes failures easier to attribute.
A good trajectory is a causal record.
The minimum useful trajectory is one schema containing run, task, environment, turns[], outcome, and measurements. Together, those blocks preserve initial conditions, ordered actions, observations, state changes, decision summaries, artifacts, criterion-level feedback, costs, safety events, and termination.
With them, a researcher can replay the work, diagnose failure, compare strategies, and design better learning signals. Without them, the record is merely a transcript: readable, but too incomplete to explain the behavior that produced it.
View all nine essays