LlamaParse 发布 ExtractBench：长文档让商业模型召回崩到 35%；自家 Agentic Plus 95.6% 登顶，成本更低。
LlamaParse 团队发布 36 页 ArXiv 论文，介绍企业文档抽取基准 ExtractBench。该基准用 370 份企业文档、4869 页、67 种文档类型评估 14 套抽取系统。核心发现是：超过 50 页的长文档会让商业 VLM 召回率跌到 35% 以下，原因是静默截断表格行。ExtractBench 用价值准确率、空间溯源和单页成本等指标做确定性评测，不依赖 LLM 裁判。同时发布的 LlamaParse Agentic Plus 在榜单上取得 95.6% 价值准确率，成本不到最接近竞品的三分之一。
We wrote a 36-page ArXiv whitepaper on ExtractBench 🧑🔬 , our effort to create the most comprehens...
We wrote a 36-page ArXiv whitepaper on ExtractBench 🧑🔬 , our effort to create the most comprehensive, schema-guided, real-world document extraction benchmark. It’s extremely detailed and covers everything from comparisons with related work on document extraction, to the dataset construction / how ground-truth is generated, to our experiments over 14+ extraction systems. Here are some of the most salient points from the paper: ✅ The benchmark scores schema-guided extraction on real enterprise documents. Given input doc + schema and predicted output from an extractor, the benchmark measures value accuracy, grounding, tags for each “challenge”, and cost. ✅ Compared to other benchmarks, we have more schemas, more evaluation dimensions, and more data domain diversity ✅ The ground-truth is constructed according to 3 doc subtypes: real-docs use a model ensemble + human review, synthetic long lists have ground-truth by construction, and scanned forms also have human review incl. boxes ✅ Our three modes (LlamaParse cost-effective, agentic, and agentic plus) are at the Pareto frontier of accuracy and cost. The full ArXiv paper is here arxiv.org/pdf/2607.29677 zr Our site extractbench.ai c3 Jerry Liu @jerryjliu0 Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex enterprise documents. The latest models are pushing the frontier of coding and knowledge work, but surprisingly they still struggle on complex doc extraction tasks in production. A well-tuned extractor must parse multi-page filings without dropping rows, emit exact spatial citations for auditability, and handle messy scans. Also they must do all of this at a viable per-page cost so that you can scale this to millions of docs in production (you can’t be paying upwards of $1 in tokens per page!) Existing extraction benchmarks fall short: they are not large/diverse enough in document domain (finance, energy, gov, auto), elements (long records, scans, grounding), and schemas. So our applied research team built ExtractBench. We evaluated 14 systems: frontier VLMs, coding agents, and specialized extraction APIs, against 370 enterprise documents: 4,869 pages, 67 document types. Our biggest finding 🧪: Short documents mask critical system flaws. On files past 50 pages, commercial VLMs collapse below 35% recall due to silent list truncation. They hold high precision, but lose output attention and drop most of the table rows. ExtractBench evaluates value accuracy, long-record completeness, spatial grounding, and per-page cost with zero LLM judges. It is 100% deterministic and reproducible. In tandem with ExtractBench, we’re also introducing 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗣𝗹𝘂𝘀, a new Extract tier in LlamaParse that debuts at #1 on the leaderboard: 95.6% value accuracy, at less than a third the cost of the closest peer. Explore the findings, download the dataset, or run the harne llamaindex.ai/blog/introduci… o/tsXfovK github.com/run-llama/Extr… o/auzmOGgUl3 H huggingface.co/datasets/llama… o/1eKisSBrE8 We will be actively evolving both our extraction benchmark as well as our extraction harness over time. If you check out either ExtractBench or LlamaParse, let us know your feedback! Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 4 🔄 3 ❤️ 25 👀 1913 📊 8 ⚡