One-sentence Summary

ExtractBench, a schema-guided enterprise document extraction benchmark spanning 4,869 pages, 370 enterprise documents, 8 business domains, and 67 document types, uses a curation pipeline that combines independent-system agreement, adjudication, known synthetic values, and human verification; it reports order-insensitive value F1, record completeness, word- and page-level grounding F1, and measured cost, with LlamaExtract Agentic Plus ranking first on all three F1 metrics at 8.1¢/page versus coding agents' 16.2-27.8¢/page.

Key Contributions

• ExtractBench is introduced as a challenge-tagged benchmark for schema-guided extraction with 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, where tags differentiate task and perception difficulty.
• A scalable schema and ground-truth curation pipeline combines independent-system agreement and adjudication for real documents, known values for synthetic lists, and human verification for forms.
• The evaluation jointly measures value accuracy via order-insensitive value F1, record completeness at scale, grounding via word- and page-level F1, and cost; LlamaExtract Agentic Plus ranks first on all three metrics at 8.1 cents per page versus 16.2 to 27.8 cents per page for coding agents, while grounding remains the clearest area for improvement.

Introduction

Extracting structured data from business documents such as invoices, claims, and regulatory filings is a high-volume, repetitive workflow where errors can be costly, and enterprises increasingly want LLM-based agents to handle schema-guided extraction at scale. Prior benchmarks often fix the extraction ontology in advance or cover only narrow dimensions such as schema complexity, long-list completeness, or long reports, while failing to jointly measure grounding, cost, scanned or handwritten inputs, and visual traceability. The authors introduce ExtractBench, a challenge-tagged benchmark of 370 documents and 4,869 pages across 8 business domains and 67 document types, designed to evaluate correct and complete JSON extraction, source grounding, robustness across document challenges, and per-page cost. They also contribute a scalable schema and ground-truth curation pipeline and evaluate 14 methods spanning commercial vision-language models, coding agents, open-source systems, and specialized extraction APIs.

Dataset

ExtractBench is a schema-guided extraction benchmark built from real documents, synthetic long lists, and scanned forms. Each document is tagged across five independent axes so errors can be traced to task difficulty, capture quality, table structure, length, or domain.

• Task definition:

  • Input is a full document, born-digital or scanned, plus a JSON Schema.
  • The schema defines fields, types, natural-language descriptions, nested objects, arrays, nullability, and value constraints.
  • Output is schema-valid JSON with source page and bounding box evidence for each value.
  • Missing document information must be returned as null.

• Dataset composition and scale:

  • Covers 8 business domains and 67 document types:
    • D1: finance and fund holdings
    • D2: energy-sector regulatory forms
    • D3: government procurement and customs
    • D4: auto valuation
    • D5: supply-chain and transactional documents
    • D6: healthcare remittance
    • D7: legal and bankruptcy filings
    • D8: real-estate closing disclosures
  • Length buckets:
    • L1: up to 10 pages
    • L2: 11 to 50 pages
    • L3: more than 50 pages
  • Perception tags:
    • P1: rotated or image-only capture
    • P2: scanned page images
    • P3: handwriting
  • Table structure tags:
    • S1: merged or hierarchical headers
    • S2: header not above its data
    • S3: table continuing across pages
    • S4: table with more than 1,000 rows
    • S5: table packed inside a single cell
  • Task challenge tags:
    • T1: long-list completeness
    • T2: needle-in-haystack, with a median of 1.6 requested fields per page
    • T3: dense documents
    • T3.a: the most common dense-form case
    • T3.e: schemas with more than 150 leaf fields

• Sources and filtering rules:

  • Real documents:
    • Schema is drafted from sample documents.
    • Multiple extraction systems from different model and pipeline families run against the candidate schema.
    • A value becomes candidate ground truth only when all systems agree, including null for absent fields.
    • Disagreements are classified by cause:
      • If more than one reading is defensible, the schema is considered ambiguous and is tightened with aliases, format requirements, location hints, and do-not-confuse guidance until reruns converge.
      • If only one reading is defensible, it is treated as a model failure and a reviewer settles the contested cells against the page.
  • Synthetic long lists:
    • Built data-first from real filings such as fund schedules, holdings registers, or creditor matrices.
    • Records are parsed verbatim or generated in the same style.
    • Rendering code creates a PDF matching the original layout, with page breaks placed by measurement.
    • Ground truth is exact by construction because every value, page, and word-level box is known from the render.
    • Mechanical checks catch style mismatches and clipping.
    • An extraction-system pool audits the finished document for rendering bugs.
  • Scanned forms:
    • Schema is authored against the blank form template and frozen before labeling.
    • An ensemble of up to five systems votes on every schema leaf.
    • Contested votes go to an adjudication agent that must inspect the page.
    • A human annotator accepts, edits, nulls, or redraws each proposed box.
    • This produces 169 human-verified documents.
    • 84% of verified fields carry a human-placed box; the rest are mostly blank fields.
  • Degraded scans:
    • Created by re-capturing clean documents.
    • No new annotation is required because the clean document ground truth carries over.

• Metadata and ground-truth construction:

  • Real-document values are confirmed by agreement across independent systems.
  • Synthetic values and boxes are exact by construction.
  • Scanned-form values and boxes are checked by a person.
  • Values are scored everywhere.
  • Box-level grounding is scored only where boxes are verified.

• Usage in the paper:

  • The provided excerpt describes ExtractBench as an evaluation benchmark for schema-guided extraction.
  • It does not state training splits or mixture ratios.
  • The paper emphasizes evaluation and error attribution through the independent task, perception, table, length, and domain axes.

Method

To properly evaluate schema-guided extraction systems, the extraction task must be well specified, requiring coherent schemas and clear expected values for every field. Creating these schemas and corresponding ground truth manually is prohibitively labor-intensive and costly, especially for long and dense documents. Relying on a single extractor output would introduce bias and repeat its mistakes. To address this, the authors design a scalable pipeline that produces high-quality schema and ground-truth pairs without requiring full manual annotation. This methodology combines three distinct document sources, each annotated using the method best suited to its characteristics: frontier-model ensembles for real documents, programmatic generation for synthetic long lists, and human labelers for scanned forms.

The overall framework for constructing ground truth across these three sources is illustrated below.

For real documents, the authors leverage a multi-stage process to establish ground truth. A schema discovery agent first drafts a schema from sample PDFs, including aliases and do-not-confuse guidance. This candidate schema is then evaluated by a diverse model pool comprising various frontier models and coding agents. Values on which all systems agree become candidate ground truth. Disagreements are diagnosed to determine their cause. If multiple readings are defensible, the schema description is tightened until reruns converge. If only one reading is defensible, it is treated as a model failure, and a human reviewer settles the contested cells against the page. The resulting ground truth includes values and page numbers, but no word boxes.

For synthetic long lists, the authors employ a programmatic generation approach to avoid human labeling entirely. The process works backwards from the data to the document. First, records are parsed verbatim from a real filing. Simultaneously, a coding agent studies the layout fonts, columns, and page chrome to generate style code. This code renders the records into a PDF that closely matches the original. Because every value is known before the PDF exists, and page and word-level boxes are read back from the render, the ground truth remains exact regardless of list length. Mechanical checks ensure style matching and prevent clipping, while a model pool audit exposes any rendering-code bugs, which are fixed before the document family is finalized. This pipeline yields ground truth with values, word boxes, and page numbers, trusted by construction and model-pool audit.

For scanned forms, human verification is essential due to unclear handwriting and ambiguous marks. The schema is authored against the blank form template and frozen before labeling. An ensemble of up to five systems votes on every schema leaf, and an adjudication agent inspects contested votes. A designated pipeline proposes a box per field, which a human annotator then accepts, edits, nulls, or redraws. This rigorous process produces human-verified documents where the ground truth includes values, word boxes, and page numbers, trusted through model votes and human verification.

The three pipelines back their ground truth differently. Real-document values are confirmed by agreement across independent systems, synthetic values and boxes are exact by construction, and form values and boxes are checked by a person. This determines which metrics each document supports, with values scored everywhere and box-level grounding applied only where boxes are verified.

Experiment

The experiments evaluate fourteen extraction systems across visual language models, coding agents, and specialized APIs using value accuracy and source-grounding metrics. The quality-cost analysis shows that specialized APIs, particularly LlamaExtract tiers, span the best tradeoff frontier, with coding agents achieving high accuracy at much higher cost and commercial VLMs staying low cost but below stronger accuracy. Breakdowns by document length, task type, perception challenges, table structure, and business domain reveal that long documents, dense schemas, scanned pages, and very large tables cause recall failures and system-specific weaknesses. Grounding remains a clear gap, since most systems provide no evidence by default and even the best systems struggle to connect values to exact word-level sources.

Fixed-ontology benchmarks cover real documents but use fixed schemas and provide only partial coverage of long records, scans, or handwriting. Schema-guided benchmarks vary in emphasis, with some focusing on schema complexity and others on long-record completeness, but they do not jointly evaluate grounding and cost. ExtractBench is distinct in combining long-record completeness, real scans and handwriting, page- and word-level grounding, and measured cost. Fixed-ontology benchmarks use fixed schemas and generally offer only partial coverage of long records and scans or handwriting. Schema-guided benchmarks evaluate either a small set of shared schemas or per-document schemas, limiting coverage of schema transfer across varied documents. ExtractBench is the only benchmark that jointly evaluates long-record completeness, real scans and handwriting, word- and page-level grounding, and measured cost.

Systems differ most on long documents and structurally complex inputs such as enormous tables. The strongest system remains above 90 F1 across document lengths and perception challenges, while many others show steep drops, especially commercial VLMs on long documents and several systems on very large tables. Some perception failures are system-specific rather than universal. Long documents cause a widening performance spread, with commercial VLMs dropping far below their short-document scores while a few systems remain close to their short-document performance. Enormous tables create the sharpest separation, with several systems returning only a small fraction of records and top systems maintaining high accuracy. Perception challenges expose blind spots that vary by system, such as relative weakness on scanned pages or rotated image-only inputs.

Grounding performance splits sharply by granularity: page-level grounding is much stronger than word-level grounding for the reported extraction systems. Long documents often reduce page-level grounding, with some systems falling to zero while a few remain more stable. Even the strongest word-level grounding remains below half, indicating that exact evidence localization is still an open challenge. Page-level grounding is consistently higher than word-level grounding across every listed system. LlamaExtract Agentic Plus leads overall at both grounding levels, but its word-level F1 is far below its page-level F1. Datalab and Extend Max Context both drop to zero page-level grounding on long documents, while Reducto Deep Extract remains relatively stable. Systems that do not return evidence by default score zero at both grounding levels. The best overall word-level grounding F1 is only 46.4%.

ExtractBench is positioned as a benchmark that combines long-record completeness, real scans and handwriting, page- and word-level grounding, and measured cost, unlike fixed-ontology or schema-guided alternatives that offer only partial coverage. Performance differences across systems become most pronounced on long documents and very large tables, with several commercial vision-language models and extraction systems degrading sharply while the strongest systems remain relatively stable. Grounding results show that page-level evidence localization is consistently much stronger than word-level localization, and long documents often cause page-level grounding to fail for some systems. Even the best word-level grounding remains low, indicating that precise evidence localization is still an open challenge.