LlamaIndex Launches ExtractBench Pro for Long-Tail Document Extraction
The shift to perfect-document scoring changes how extraction systems should be evaluated: one wrong cell fails the whole doc, which mirrors the economics of enterprise workflows where partial accuracy is often unusable. Teams building document pipelines can use the benchmark and its synthetic template pipeline to stress-test models before deployment without exposing private data. For comparison, its hardest cases sit well above what saturated benchmarks like ExtractBench already measure.
LlamaIndex introduced ExtractBench Pro, a benchmark for real-world document extraction built through a synthetic pipeline that mines real document templates while varying data, template layout and rendering. It targets a long tail of complex use cases the company sees from customers, including trial balances, well logs, canvass binders, site plan reviews, duty reconciliations and loss runs. The benchmark measures conventions, internal jargon density, aggregation reasoning, drawn relations, charts and long chart sheets. Its headline metric is the proportion of documents achieving a perfect 100% value f1 score, alongside value f1 and grounded f1. LlamaIndex notes existing extraction benchmarks are saturated, with its cheapest modes scoring 93%+ on ExtractBench.
One of the most important challenges in advancing enterprise AI is creating evaluation benchmarks that reflect their real data and workflows - without using their actual data or workflows due to privacy/security restrictions We see our customers upload a massive long tail of complex documents for extraction, which break even the most complex frontier models. In order to create a representative benchmark that adheres to this complexity, we invested heavily in a synthetic pipeline that can mine real document templates, but allow flexibility for variation of not only the data, but the template and rendering. Come check out ExtractBench Pro! llamaindex.ai/blog/introduci… Website: extractbenchpro.ai Your browser does not support the video tag. 🔗 View on Twitter Jerry Liu @jerryjliu0 Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction. We see a massive long tail of document complexity in terms of use cases brought on from our customers, that are completely unsolved from a price-performance standpoint. These include trial balances, well logs, canvass binders, site plan reviews, duty reconciliations, loss runs, and more. Most existing extraction benchmarks are saturated by the frontier models (our cheapest modes now score 93%+ on ExtractBench). So there is enormous economic value in building a complex extraction benchmark that reflects this long-tail of knowledge work. We spent a long time curating the most complex elements that we saw from real world extraction use cases: ✅ Document conventions: containing legends or rules where the rest of the document should be interpreted ✅ Density of internal jargon: specific definitions that must be cross-referenced within a doc ✅ Aggregation and reasoning: beyond returning exact values on a page, infer values derived from aggregating multiple sources of information ✅ Drawn relations: complex diagrams like procedures and circuits ✅ Charts inside documents: read precise points within a chart ✅ Long chart sheets: some data series like well logs cram all this data into an extremely long page Within the industries where document extraction is relevant, even small mistakes can have enormous negative impacts. Therefore our headline evaluation metric is based on perfect documents: the proportion of docs with perfect (100%) value f1 score. (Besides this we also measure value f1 and grounded f1) It’s an extremely hard task for even the latest frontier models to solve, topping out at ~58% or so in Value F1 score for Codex with Astra - which is quite good but extremely expensive (almost $1 per page!) Our LlamaParse agentic/agentic plus modes are 20-40% and cover the left side of the Pareto frontier, at price ranges that are feasible for full production use. ExtractBenchPro is available today as a website/technical blog, and also on Github and HuggingFace. Check it out! Blog: llamaindex.ai/blog/introduci… Website: extractbenchpro.ai Hugging Face: huggingface.co/datasets/llama… Github (is ExtractBench): github.com/run-llama/Extr… Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 9 🔄 2 ❤️ 16 👀 1123 📊 10 ⚡