Jerry Liu 分享文档解析基准设计:ParseBench-2 与 DPP 的颗粒度实践
LlamaIndex 的 Jerry Liu 把文档解析评测拆成 6 维 330 场景,还配了 516 条 LLM 判分规则。正在做 RAG 或解析管道的人可以直接抄这套加权评估思路,比只看一个总分靠谱多了。
LlamaIndex 创始人 Jerry Liu 发文讨论文档解析基准的核心难题:如何让评估同时具备颗粒度和人类偏好对齐。他给出两条互补路径:确定性指标方面,ParseBench-2 将解析拆成 6 个维度、330 个测试场景,按人工定义的严重度加权——数字错误扣 10 分以上,多余 HTML 标记扣 0 分。LLM as a judge 方面,其 Document Parsing Protocol(DPP)定义了 140 条规则、516 项书面检查,每项由专门的 judge prompt 执行。
颗粒度带来的价值在于可以精确编码人类偏好,并按需对图表错误、表格错误、表单错误、版式错误等维度加权。Jerry Liu 也坦承 ParseBench-2 和 DPP 都不完美,表示后续会持续迭代。对正在选型文档解析方案或自建评估体系的开发者来说,这套分维度加权的思路可直接借鉴。
One of the hardest parts about building any benchmark that measures high-dimensional output is making it granular, while also being aligned with human preferences. This especially holds true for document parsing. Granularity is helpful because it lets you more precisely encode human preferences. Within document parsing, there are many categories of errors across various document dimensions: chart errors, table errors, form errors, layout errors, and more. Granularity also enables customization - it lets you better choose the dimensions you care about the most and weigh them more heavily. You can have a granular benchmark using either deterministic metrics or LLM as a judge; they’re not mutually exclusive: 1️⃣ Deterministic: ParseBench-2 splits parsing into 6 dimensions and 330 test scenarios. Each dimension is weighed by human-defined severity (a wrong digit should cost 10+ pts, extra HTML markup costs 0). 2️⃣ LLM as a judge: You can also define granular metrics where each one is measured by LLM as a judge. Our Document Parsing Protocol (DPP) has a comprehensive taxonomy of 140 rules / 516 written checks, each codified by a judge prompt! Building a benchmark that is granular but aligned is challenging. There’s no free lunch; neither ParseBench-2 nor DPP are ‘perfect’, but they do make an honest effort in advancing every criteria. We’re shipping a lot more iterations here. Keep your eyes peeled! ParseBench-2: parsebench.ai Document Parsing Protocol: parseprotocol.com
This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include ParseBench-2, LlamaIndex, document parsing, RAG.
Source attribution: curated from Jerry Liu (@jerryjliu0).