LlamaIndex 搞了个提取基准，测了 14 个系统发现长文档上 VLM 会悄悄丢行，还顺带出了个便宜三倍的提取服务，搞文档处理的值得看看。
LlamaIndex 推出 ExtractBench，号称最全面的复杂企业文档信息提取基准。该基准测试了 14 个系统，包括前沿 VLM、编码智能体和专用提取 API，覆盖 370 份企业文档、4,869 页和 67 种文档类型。最大发现是超过 50 页的文档中，商业 VLM 因静默列表截断，召回率跌破 35%。ExtractBench 评估值准确率、长记录完整性、空间定位和每页成本，完全确定性且无需 LLM 评判。同时发布 LlamaParse 新层级 Agentic Plus，以 95.6% 值准确率位居榜首，成本仅为最接近对手的三分之一。
Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex e...
Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex enterprise documents. The latest models are pushing the frontier of coding and knowledge work, but surprisingly they still struggle on complex doc extraction tasks in production. A well-tuned extractor must parse multi-page filings without dropping rows, emit exact spatial citations for auditability, and handle messy scans. Also they must do all of this at a viable per-page cost so that you can scale this to millions of docs in production (you can’t be paying upwards of $1 in tokens per page!) Existing extraction benchmarks fall short: they are not large/diverse enough in document domain (finance, energy, gov, auto), elements (long records, scans, grounding), and schemas. So our applied research team built ExtractBench. We evaluated 14 systems: frontier VLMs, coding agents, and specialized extraction APIs, against 370 enterprise documents: 4,869 pages, 67 document types. Our biggest finding 🧪: Short documents mask critical system flaws. On files past 50 pages, commercial VLMs collapse below 35% recall due to silent list truncation. They hold high precision, but lose output attention and drop most of the table rows. ExtractBench evaluates value accuracy, long-record completeness, spatial grounding, and per-page cost with zero LLM judges. It is 100% deterministic and reproducible. In tandem with ExtractBench, we’re also introducing 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗣𝗹𝘂𝘀, a new Extract tier in LlamaParse that debuts at #1 on the leaderboard: 95.6% value accuracy, at less than a third the cost of the closest peer. Explore the findings, download the dataset, or run the harne llamaindex.ai/blog/introduci… o/tsXfovK github.com/run-llama/Extr… o/auzmOGgUl3 H huggingface.co/datasets/llama… o/1eKisSBrE8 We will be actively evolving both our extraction benchmark as well as our extraction harness over time. If you check out either ExtractBench or LlamaParse, let us know your feedback! Your browser does not support the video tag. 🔗 View on Twitter LlamaIndex 🦙 @llama_index Introducing 𝗘𝘅𝘁𝗿𝗮𝗰𝘁𝗕𝗲𝗻𝗰𝗵: the most comprehensive benchmark for information extraction from complex enterprise documents. Our applied research team tested: 14 systems — frontier VLMs, coding agents, extraction APIs — on 370 enterprise docs, 4,869 pages, 67 doc types. Zero LLM judges, fully deterministic. Biggest finding: past 50 pages, commercial VLMs collapse below 35% recall. Precision stays high, but they silently drop most of the table rows. What is your extraction agent missing? Run ExtractBench to see tod llamaindex.ai/blog/introduci… o/5UE8frC github.com/run-llama/Extr… o/27vJ36bL54 H huggingface.co/datasets/llama… o/oeGlwPjcQh Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 6 🔄 3 ❤️ 13 👀 859 📊 7 ⚡
- LlamaIndex13:00原文