LlamaIndex 发布 ParseBench-2：可按业务偏好自定义权重的文档解析评测框架

其核心机制是：不再硬编码单一确定性指标，而是让用户先用严重度策略描述每类解析错误的业务影响（类似在高维空间拟合线性回归），再由编程 Agent 自动生成并优化与这些偏好对齐的评测指标。相比简单的一次性 LLM-as-a-judge 调用，这种方式能产出更细粒度、可复现、可量化的分维度得分，且可大规模运行。

与现有方案对比：OmniDocBench 依赖脆弱的确定性指标，ParseArena 的 LLM 评审则较粗糙且带随机性。ParseBench-2 面向不同场景可自定义侧重——后台数据场景看重表单、边界框和扫描件，投研报告场景更看重表格和图表。仓库、博客和技术报告全部开源。

You shouldn’t blindly trust benchmarks - you should be able to encode your preferences relative to your use cases and design eval metrics around that. We’re excited to release ParseBench-2 as not only a comprehensive document parsing benchmark, but also a framework for parsing benchmark design. Where you can articulate a set of human preferences of what “good parsing” looks like and “hillclimb” evaluation metrics against that. The important part is not hand-crafting a deterministic parsing metric. Metric design can be automated by coding agents; what’s important is an open framework + datasets for encoding human preferences and optimizing metrics. 📋 For instance, if you’re dealing with mostly backoffice data, you may care about accurate form parsing, bounding boxes, and scans! 📊 On the other hand, if you’re dealing with investor research reports, you may care more about tables and charts, but less around handwriting. We let you encode a severity policy that encodes the impact of every negative parsing error, and forces the metric to fit against it - think of fitting a linear regression in high-dimensional space. A baseline approach for encoding human preferences is LLM as a judge. But the result of this approach vs. a “simple” LLM as a judge call is that you get much more granular, repeatable, quantifiable breakdowns of each dimension that are efficiently run at scale. For instance, one OCR tool might be better at linearizing text, but worse at preserving formatting. Another OCR tool might be better at preserving empty rows in tables but struggle with merged cells. The scores reflected on our website do reflect a “default” preference, but it is explicitly and transparently articulated both in our Github and technical report. The important part is that you can change and customize it for your needs. ParseBench-2 is our most ambitious, comprehensive open benchmark to date. It is not simply a set of deterministic metrics, but a framework for designing parse evals. We’re rapidly working on improvements and expansions (e.g. fixing minor annotation issues and tuning metrics). If you have thoughts please chime in! Everything is fully OSS. ParseBench-2 Repo github.com/run-llama/pars… ZY ParseBench-2 Blog llamaindex.ai/blog VB Technical Report llamaindex.ai/files/parseben… me

This report is categorized under AITOP's "Products" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include ParseBench-2, LlamaIndex, document parsing, RAG.

Source attribution: curated from Jerry Liu (@jerryjliu0).