LlamaIndex做了个ExtractBench，测了14个文档提取系统，发现Codex和OCR各有短处，他们自己的Agentic Plus全场景最稳。
LlamaIndex本周发布文档提取基准ExtractBench，覆盖1950年代监管文件、手填税表、传真阈值化、复印件色调、传感器噪声及手机拍摄等真实场景。该基准测试了14个系统，发现各系统的感知盲区互不重叠。Codex在扫描和手写文档上准确率超过93%，但旋转页面跌至约80%。专用OCR API在旋转和手写上表现稳定，扫描场景只有81%。Gemini 3.5 Flash在扫描页上从88.6%降至71.1%。LlamaIndex的Agentic Plus是唯一无盲区的系统，在旋转、扫描、手写上的准确率分别为95.9%、93.9%、93.8%，波动仅2个百分点。
There's a lot of real-world documents that are scanned, rotated, handwritten, or some combination of...
There's a lot of real-world documents that are scanned, rotated, handwritten, or some combination of any of these elements. This week we created a comprehensive document extraction benchmark that contains documents tagged with various "perception challenges", along with other tags denoting task challenges, table structure, business domain. These docs include regulatory filings, hand-filled tax forms, photocopied docs, sensor noise, and more. Codex is surprisingly good at scans, but not great on rotated docs. OCR solutions are reasonable on rotations/handwriting but struggle on more general scans. Check out ExtractBench! ArXiv: arxiv.org/pdf/2607.29677 Site: extractbench.ai LlamaIndex 🦙 @llama_index Every document extraction system has a perception blind spot. We mapped them. For ExtractBench, we tested 14 systems on documents that weren't born digital: 1950s regulatory filings, hand-filled tax forms, and pages degraded with fax thresholding, photocopier tone curves, sensor noise, and phone-camera capture. The failures don't overlap. 💡 Codex reads scans and handwriting above 93%, then drops to ~80% on rotated or image-only pages. 💡 Specialized APIs are the exact inverse: fine on rotation and handwriting, 81% on scans 💡 Gemini 3.5 Flash falls from 88.6% to 71.1% the moment a page is scanned. You benchmark on clean PDFs. Production sends you a shadowed photocopy from 1953. Our new Extract Tier, Agentic Plus, was the only system with no blind spot: 95.9% / 93.9% / 93.8% across rotated, scanned, and handwritten, a 2-point spread where others swing 10+. Learn more about ExtractBench 👇️ Bl lnkd.in/gNm97fXp ceB6 Pap lnkd.in/euAfScWx kQpF Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 0 🔄 0 ❤️ 3 👀 412 📊 1 ⚡
- LlamaIndex08-13 14:05原文