LlamaParse搞了个ExtractBench，测出来商业VLM处理长文档会漏数据；他们新出的Agentic Plus准确率95.6%，成本还低。做文档提取的可以看下。
ExtractBench是面向复杂企业文档提取的基准，包含370份文档、4869页和67种文档类型。它评估了14个系统，包括前沿VLM、编程智能体和专用提取API。结果显示，超过50页的文档中，商业VLM的召回率低于35%，主要问题是静默截断列表。LlamaParse新推出的Agentic Plus层级在基准上达到95.6%的值准确率，成本不到最接近竞争对手的三分之一。
We think a lot about the distribution of all real-world documents, particularly for real-world use c...
We think a lot about the distribution of all real-world documents, particularly for real-world use cases. We think about the distribution in terms of page length, field length, domain, and task type. Some documents have a single page but can have anywhere from 1-10 fields to 500-1000 fields to be extracted. Other documents are hundreds of pages long and can have 10k-100k fields. On long documents (e.g. a bankruptcy creditor matrix), commercial VLMs lose attention and create large swaths of missing or hallucinated values. The number of fields to extract per page also depends on the extraction task. 1. Needle-in-a-haystack: You find very specific information within a general document corpus. Generally 1-10 fields per page 2. Dense documents: typically extract from filled forms. Quite dense per page, though not super long in terms of total # pages. 3. Long-lists: Documents with both massive #'s of values per page (e.g. 100-1k fields per page) and massive numbers of pages. We'd love for you to explore our overall data taxonomy within ExtractBench, our comprehensive benchmark for real-world document extraction tasks. I love the diagram we show below in Figure 6, that offers one slice of the diverse set of data that we curate! Come check out the ArXiv paper: arxiv.org/pdf/2607.29677 ExtractBench: extractbench.ai Jerry Liu @jerryjliu0 Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex enterprise documents. The latest models are pushing the frontier of coding and knowledge work, but surprisingly they still struggle on complex doc extraction tasks in production. A well-tuned extractor must parse multi-page filings without dropping rows, emit exact spatial citations for auditability, and handle messy scans. Also they must do all of this at a viable per-page cost so that you can scale this to millions of docs in production (you can’t be paying upwards of $1 in tokens per page!) Existing extraction benchmarks fall short: they are not large/diverse enough in document domain (finance, energy, gov, auto), elements (long records, scans, grounding), and schemas. So our applied research team built ExtractBench. We evaluated 14 systems: frontier VLMs, coding agents, and specialized extraction APIs, against 370 enterprise documents: 4,869 pages, 67 document types. Our biggest finding 🧪: Short documents mask critical system flaws. On files past 50 pages, commercial VLMs collapse below 35% recall due to silent list truncation. They hold high precision, but lose output attention and drop most of the table rows. ExtractBench evaluates value accuracy, long-record completeness, spatial grounding, and per-page cost with zero LLM judges. It is 100% deterministic and reproducible. In tandem with ExtractBench, we’re also introducing 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗣𝗹𝘂𝘀, a new Extract tier in LlamaParse that debuts at #1 on the leaderboard: 95.6% value accuracy, at less than a third the cost of the closest peer. Explore the findings, download the dataset, or run the harne llamaindex.ai/blog/introduci… o/tsXfovK github.com/run-llama/Extr… o/auzmOGgUl3 H huggingface.co/datasets/llama… o/1eKisSBrE8 We will be actively evolving both our extraction benchmark as well as our extraction harness over time. If you check out either ExtractBench or LlamaParse, let us know your feedback! Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 0 🔄 0 ❤️ 6 👀 1009 📊 1 ⚡