做文档提取的可以看看，LlamaIndex搞了个大基准，测了14个模型，发现长文档上大模型掉链子，还顺带出了个便宜又准的提取工具。
LlamaIndex推出ExtractBench，一个覆盖4869页、67种文档类型、8个真实领域的企业文档提取基准。该基准包含超过1k行的表格、嵌套表格、跨页表格、扫描件和手写文档等复杂情况。评测了14个系统，包括前沿VLM、编码智能体和专用提取API。关键发现是超过50页的文档中，商业VLM因静默列表截断导致召回率低于35%。同时发布LlamaParse新层级Agentic Plus，以95.6%的值准确率位居榜首，成本仅为最接近对手的三分之一。
ExtractBench is one of the most comprehensive benchmarks for real-world document extraction. ✅ It ...
ExtractBench is one of the most comprehensive benchmarks for real-world document extraction. ✅ It covers 4869 pages, across 67 document types, spanning 8 real-world domains: finance, energy, gov, auto, supply chain, healthcare, legal, real estate ✅ It covers a distribution of short, medium, and long documents ✅ It covers a variety of very complex table edge cases: tables with over 1k rows, nested tables within cells, cross-page tables, and more ✅ It covers scans, handwriting, and rotated pages We benchmarked across 14 different VLMs, coding agents, and document extraction APIs. Everything is fully public on our blog, ArXiv, Github, and HuggingFace: Blog: llamaindex.ai/blog/introduci… Hugging Face: huggingface.co/datasets/llama… Github: github.com/run-llama/Extr… ArXiv: arxiv.org/pdf/2607.29677 Jerry Liu @jerryjliu0 Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex enterprise documents. The latest models are pushing the frontier of coding and knowledge work, but surprisingly they still struggle on complex doc extraction tasks in production. A well-tuned extractor must parse multi-page filings without dropping rows, emit exact spatial citations for auditability, and handle messy scans. Also they must do all of this at a viable per-page cost so that you can scale this to millions of docs in production (you can’t be paying upwards of $1 in tokens per page!) Existing extraction benchmarks fall short: they are not large/diverse enough in document domain (finance, energy, gov, auto), elements (long records, scans, grounding), and schemas. So our applied research team built ExtractBench. We evaluated 14 systems: frontier VLMs, coding agents, and specialized extraction APIs, against 370 enterprise documents: 4,869 pages, 67 document types. Our biggest finding 🧪: Short documents mask critical system flaws. On files past 50 pages, commercial VLMs collapse below 35% recall due to silent list truncation. They hold high precision, but lose output attention and drop most of the table rows. ExtractBench evaluates value accuracy, long-record completeness, spatial grounding, and per-page cost with zero LLM judges. It is 100% deterministic and reproducible. In tandem with ExtractBench, we’re also introducing 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗣𝗹𝘂𝘀, a new Extract tier in LlamaParse that debuts at #1 on the leaderboard: 95.6% value accuracy, at less than a third the cost of the closest peer. Explore the findings, download the dataset, or run the harne llamaindex.ai/blog/introduci… o/tsXfovK github.com/run-llama/Extr… o/auzmOGgUl3 H huggingface.co/datasets/llama… o/1eKisSBrE8 We will be actively evolving both our extraction benchmark as well as our extraction harness over time. If you check out either ExtractBench or LlamaParse, let us know your feedback! Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 2 🔄 2 ❤️ 13 👀 1385 📊 5 ⚡
- LlamaIndex08-11 13:00原文