Document Extraction With LLMs: Building a Pipeline That Survives Real Documents

The Short Answer

Point a frontier model at a PDF with a JSON schema and structured outputs turned on, and you get valid JSON whose values are unreliable. On Contextual AI's ExtractBench, six frontier models produced 0% valid output on a 369-field financial reporting schema, and accuracy degraded sharply as schema breadth grew. A pipeline that survives real documents splits the schema, selects pages before extracting, parses with a document-specific model, and routes a small share of documents to a human with page-level provenance attached.

Why the Default Architecture Fails

The 2026 default is a single call. PDF in, JSON schema attached, structured outputs on, extraction out. It breaks in two places, and neither is where teams look.

The first is the page. MMLongBench-Doc, a benchmark of 1,062 expert-annotated questions over 130 PDFs averaging 49.4 pages, tested 14 vision-language models. The best of them, GPT-4o, reached an F1 of 42.7%, and the runner-up GPT-4V reached 31.4%. The result that should change your architecture is the other one: 12 of the 14 models performed worse reading document screenshots than their own LLM counterparts did reading lossy OCR text of the same pages.

The second is the schema. ExtractBench, published by Contextual AI in February 2026, pairs 35 human-validated PDFs with JSON Schemas across five schemas and four domains: finance, academia, hiring, and sports. It scores 12,867 evaluatable fields. Across GPT-5 and 5.2, Gemini 3 Flash and Pro, and Claude Opus 4.5 and Sonnet 4.5, performance degraded sharply with schema breadth and bottomed out at 0% valid output on a 369-field financial reporting schema, across every model tested.

Split the Schema Before You Tune the Prompt

ExtractBench's schemas span tens to hundreds of fields, and that axis is the one that predicts failure. At the narrow end, frontier models produce usable output. At 369 fields, every model tested produced nothing valid at all.

That is not graceful degradation. Past a certain width the model stops emitting usable output entirely, retries do not recover it, and an automatic retry loop converts a failed document into a cost incident.

Decision rule: decompose any schema wider than roughly 40 fields. Split by document section, run one call per sub-schema, merge, then validate the merge as a separate step. Calibrate the exact threshold on your own documents, because it moves with nesting depth and array density. The direction is not in question. Teams that skip decomposition spend a quarter prompt-engineering a schema that no model in the benchmark could satisfy.

Never Send a Long Document Whole

Long input degrades accuracy by itself, independent of retrieval quality. The paper "Context Length Alone Hurts LLM Performance Despite Perfect Retrieval" tested five open and closed models on math, question answering, and coding. Even when a model retrieved all the relevant information perfectly, accuracy fell by between 13.9% and 85% as input length grew, entirely within the models' claimed context windows. The effect survived replacing the irrelevant tokens with whitespace, and it survived masking them out so the model attended only to the relevant ones. The mitigation the authors propose is worth copying: prompt the model to recite the retrieved evidence before it answers.

Position compounds length. SynthDocBench, released by ServiceNow AI and Mila in July 2026, varies document length, layout, modality, and question type independently so failures can be attributed to a cause. Across seven frontier VLMs it found the middle third of a document hardest for five of six models, with five of six also showing a negative early-to-late trend and a steepest decline of 8.3 percentage points. Chart comprehension broke down outright in long-document settings.

Representation matters as much as page count. Amazon's Document Haystack builds 400 document variants running from 5 to 200 pages, inserting pure-text and text-plus-image needles at varying depths across 8,250 questions, which is the right way to test whether your parser survives depth. Combined with the MMLongBench-Doc finding, the practical guidance is consistent: for born-digital PDFs, feed extracted text plus layout, not screenshots. Reach for page images only when the source is genuinely scanned.

Tune Page Selection for Recall, Not Precision

Retrieval errors and aggregation errors are different problems, and only one of them is fixed by better page selection.

MMLongBench-Doc is built to separate the two. 33.2% of its questions are cross-page, requiring evidence from more than one page, and 22.8% are deliberately unanswerable to catch hallucination. Those two design choices name the two things that break multi-page extraction in production.

For the retrieval half, bias hard toward recall. A missing page makes the correct answer unreachable regardless of how good the model is. An extra irrelevant page costs some accuracy, per the context-length result above, but it costs far less than an absent one. Over-retrieve deliberately, then let the extraction step discard what it does not need.

The aggregation half does not respond to retrieval at all. A total assembled from a schedule plus a footnote comes back confidently wrong and reconciles to nothing, even when both pages sit comfortably in context. Handle that with arithmetic validation after extraction rather than with a better prompt, and give the model an explicit way to abstain. Abstention carries a price: instructions that raise correct refusals on unanswerable questions also produce refusals on answerable ones. Set that tradeoff on purpose and measure both sides of it on your own evaluation set.

The Failure Modes That Survive QA

These are the ones that ship, because none of them raises an error.

Silent truncation. The extractor reads the opening pages and a few near the end, skips the middle, and returns well-formed JSON that looks complete. It is visible only against gold labels, and it varies between runs, so single-run QA misses it. Run every test document five times and diff the outputs.

Over-generation. A vision model can emit text that is not on the page. The paper "Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models" documents the mechanism on degraded scans: the model does not register that the image is unreadable, falls back on linguistic priors, and produces fluent invented content instead of an error. Consensus across perturbed views of the same page is the practical detector, because a hallucinated reading is unstable under perturbation and a real one is not.

Table structure hallucination. Table extraction rarely fails loudly. It fails by returning a plausible table with values mapped to the wrong columns, most often where merged or spanning cells are involved. PulseBench-Tab, published in April 2026, scores table extraction with a graph-based metric precisely because cell-level string comparison hides this class of error. It comes from a vendor in the space, so read its rankings with that in mind. The failure mode it targets is real regardless of who measured it.

Language collapse found after launch. Extraction quality is not uniform across scripts, and vendor benchmarks are usually reported on English documents. PulseBench-Tab is explicitly multilingual for that reason. Validate on your actual language and layout mix before you sign a contract, not after the first non-Latin batch arrives.

Do Not Route on the Model's Confidence

The instinct is to gate the human review queue on a confidence signal: token log-probabilities, or a self-reported certainty field in the schema. Both are weak error detectors, and they fail in the direction that hurts. A model that has fallen back on linguistic priors, as in the OCR hallucination case above, is fluent and therefore confident. The queue fills with correct documents while the wrong ones pass through carrying high scores.

Route on structure instead. ExtractBench's evaluation design draws a distinction worth building into your own pipeline: omission and hallucination are different errors that need different handling. A field the model declined to fill is a retrieval or coverage problem. A field it filled wrongly is a verification problem, and only cross-checking catches it.

Decision rule: set the review threshold from the cost of a wrong field, not from a target automation rate. If a value lands in a ledger, a payment, or a clinical record, require two independent extractions to agree, or route it to a human. Plan for a straight-through rate in the mid-to-high nineties rather than 100.

Require provenance on every field: page number plus bounding box, or a verbatim source span. A pipeline that cannot show where a number came from gets resolved by re-keying the document by hand on the first dispute, and that erases the ROI case you built it on.

What It Actually Costs

Per-page OCR is the cheapest part of the system, and it is where most cost conversations stop. Amazon Textract's published US pricing, read from the vendor page on 25 August 2026:

The shape matters more than the absolute figures. Structured extraction costs ten to thirty times raw OCR at the same page count, and the other major providers price on the same pattern. Read your own vendor's current page before you build a model on any of these, since OCR list prices move.

Model-based parsing lands in a comparable range. Google's Gemini API documentation counts each PDF page as 258 tokens, so 1,000 pages is roughly 258,000 input tokens. Multiply that by your model's current published input rate and compare it against the table above. At flash-tier pricing the parse costs cents.

That is the actual point. The parse is rarely the dominant line item. Evaluation, verification, orchestration, and human review are, and those scale with your accuracy requirement rather than with your page count. Budget accordingly.

Treat Every Document as Untrusted Input

NIST's "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations" (AI 100-2e2025, March 2025) treats indirect prompt injection as a named class of attack, and its guidance is to assume injection is possible wherever a model reads untrusted input. A document you did not author is untrusted input.

An invoice carrying white-on-white text that redirects the payee is a working attack against most extraction pipelines shipped this year, because the parser reads the text layer while the human reviewer sees the rendered page. Never let extracted document text reach a tool call, a system prompt, or a downstream agent unquoted. Treat every extracted string as data, and validate payment and identity fields against your own records rather than against the document.

The Pipeline Shape That Works

1. Build the evaluation set first. Fifty to two hundred real documents with hand-labeled gold fields, covering your worst layouts, your scanned pages, and your non-Latin scripts. Nothing below this line is tunable without it.
2. Classify and triage on ingest, and route oversize files separately. The Claude API accepts a 32 MB request and up to 600 pages, dropping to 100 pages when the request's context window is under 1M tokens, and rejects password-protected PDFs. The Gemini API caps at 50 MB and 1,000 pages. Real corpora contain the 900-page loan file.
3. Parse the page with a document-specific model rather than a general one. Docling, MIT-licensed and developed at IBM Research Zurich under the LF AI and Data Foundation, is the credible self-hosted option when documents cannot leave your network.
4. Select candidate pages per sub-schema, tuned for recall.
5. Extract per sub-schema. Never one wide call.
6. Separate the reasoning pass from the JSON-emitting pass when extraction requires inference. Structured outputs guarantee that the shape and the types match your schema. They say nothing about whether the values are right.
7. Verify with cross-extraction agreement, arithmetic checks, and provenance on every field.
8. Route disagreements and abstentions to a human queue, and size that queue from what step 1 tells you rather than from a target you picked in advance.

How We Approach This at SASID

The healthcare system we built runs HIPAA-compliant retrieval over medical records, and it cut insurance denial appeals from 30 to 60 minutes of specialist time to under 2 minutes. The prompt was the least interesting part of that build. What made it usable in a clinical workflow was narrow extraction targets, citable provenance back to the source record, and a defined review path for the cases the system flagged as uncertain. Our delivery pattern is a proof of concept in days and production in 4 to 8 weeks, followed by 90 days of monitoring, because the verification layer is what takes the time and it is what decides whether the pipeline is economic.

Get a Free Technical Assessment

If you are building or fixing a document extraction pipeline, we offer a free technical assessment: a 30-minute call about your documents and your accuracy requirements, followed by a written roadmap within 48 hours covering architecture, evaluation design, throughput cost, and timeline. There is no obligation, and the roadmap is yours to keep. Book at sasid.ai.