LlamaIndex launches ParseBench-2 for severity-aware document parsing evaluation
Severity-weighted evaluation changes how teams choose parsing models: two parsers with identical edit-distance scores can differ sharply once digit flips count 25x formatting noise. Teams building RAG pipelines or financial document workflows should re-run their parsers under a policy tuned to their own error tolerance. The public taxonomy of 330+ error types also doubles as a checklist for auditing in-house parsing QA.
LlamaIndex has released ParseBench-2, an updated benchmark that evaluates document parsing quality, with a particular focus on complex table parsing. The core change over existing benchmarks like OmniDocBench, OlmOCR-Bench, and ParseBench-1 is a severity policy layer applied to deterministic metrics, letting users specify how damaging different error types are instead of relying on raw edit distance or averaged binary tests.
ParseBench-2 ships with a taxonomy of over 330 error types, a large subset covering table errors, organised into five severity levels. The most severe category includes wrong digits, misplaced decimals, swapped values, and missing columns — errors catastrophic in financial reporting. Medium severity covers issues like reordered body rows or one cell split into two, while formatting differences and HTML markup are classified as harmless. The default severity weighting for critical errors is 10, but the policy is customisable, so a finance team could raise it to 25.
The benchmark addresses a known gap: conventional metrics score harmless formatting deviations the same as sign flips on financial figures, even though downstream knowledge work treats them very differently. Details are published in a blog post and tech report on llamaindex.ai.
One of the hardest parts around document parsing is parsing complex tables. Besides the task itself being difficult, it is also hard to measure what ‘good’ looks like. Ultimately it comes down to intuition around what is a catastrophic error for downstream knowledge work. * Formatting differences (markdown vs. html tags) are harmless * Making the header row a “data” row is also probably harmless (any human/model can interpret it as the header anyways) * Swapping the signs on a number, particular in financial reporting, is much more severe. It is hard to encode what “good” looks like in a quantifiable way. Existing deterministic tests/metrics (OmniDocBench/OlmOCR-Bench/ParseBench-1) use some variant of edit distance, or an average over binary tests, making it hard to distinguish severe errors from normal ones. With our recent release of ParseBench-2, our answer is to tune any deterministic metric by a severity policy, where you can explicitly state how good or bad different errors are. * For instance, in a finance setting, dropping digits/flipping signs is really bad! Our default severity weighting is 10, but you might decide 25 - the policy is meant to be customizable by the user. * Check out the examples below highlighting this. You can compare the impact of “actually bad” or “harmless” perturbations on a few different benchmarks incl. ParseBench-2. We have a taxonomy of 330+ error types, a big subset of which is defined around table errors, categorizing them from 5 severity levels: * most severe: wrong digits, decimals misplaced, values swapped, column missing * medium severe: body rows reordered, one cell split into two, etc. * harmless: formatting errors, html markup More details around table methodology in blog/tech report. Blog: llamaindex.ai/blog/introduci… Tech report: llamaindex.ai/files/parseben…
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include LlamaIndex, ParseBench, OmniDocBench, RAG.
Source attribution: curated from Jerry Liu (@jerryjliu0).
