ProductsMulti-sourceSelected

LlamaIndex launches ParseBench-2 for document parsing evaluation

Why we picked it

The redesign addresses a real evaluation blind spot: teams choosing parsers for RAG pipelines often get scores that contradict actual output quality. Practitioners building document-ingestion workflows can use the Metric Lab to weight errors by their own priorities — critical for finance and forms-heavy use cases. Compared with static metric suites like older OCR benchmarks, this approach trades simplicity for human-aligned scoring, and teams should stress-test the trade-off before adopting it.

LlamaIndex has released ParseBench-2, an updated benchmark for evaluating document parsing systems, following the earlier ParseBench that has been cited by NVIDIA, Cohere and Perplexity. The new version spans 4,317 pages across six dimensions — tables, charts, layout, formatting, text and now forms — with full reference tables for all 1,025 charts.

Body

The key change is in how scoring works: instead of relying on static deterministic metrics or LLM-as-a-judge, LlamaIndex first decides how much each error type should matter, then tests and tunes the scoring until it agrees. They validated this with 330 perturbation types to confirm scores move by the right amount. The motivation came from a concrete failure: on a McDonald's financial statement, ParseBench-1 scored a parser that broke a dollar amount higher than one that got every number right.

The benchmark rests on six design criteria — representative data, granularity, quantifiability, repeatability, controllability and alignment with human judgment. A Metric Lab lets users set scoring priorities for their own workloads, and LlamaIndex is shipping rapid iterations on annotations and metric quality.

Original · Jerry Liu

Building a great benchmark is challenging, building a perfect benchmark is near impossible. This especially holds true in the world of document parsing. I wrote a blog post that details the core design criteria needed for a document parsing benchmark, and the historical attempts to get there - from those based on static deterministic metrics to LLM as a judge approaches. Building a “great” benchmark requires at least the following: 1. Representative data 2. Granularity 3. Quantifiable 4. Repeatable 5. Controllable 6. Aligned with human judgment Existing benchmarks based on deterministic metrics are oftentimes granular/repeatable, but not controllable or aligned with human judgment. Those based on LLM as a judge are not granular, quantifiable, repeatable, and can obscure how human judgment is encoded. Last week we launched ParseBench-2 in an attempt to tackle all of these dimensions. It contains a taxonomy of metrics across 6 dimensions (tables, charts, layout, formatting, text, and forms) that precisely measure document parsing failures and quantifies their impact. It’s certainly not perfect, but does make an honest attempt towards advancing every design criteria. It’s also designed to be customizable by the user (see our metric lab!) We’re rapidly shipping new iterations of this benchmark to improve annotations, metric quality, and customizability. If you do check it out, would love to hear your thoughts. Blog: llamaindex.ai/blog/building-… ParseBench-2: parsebench.ai LlamaIndex 🦙 @llama_index ParseBench has been cited by NVIDIA, Cohere and Perplexity. It still isn’t a perfect benchmark. Part of getting closer meant owning where ParseBench-1 got things wrong. On one McDonald’s financial statement, it scored a parser that broke a dollar amount above one that got every number right. So for ParseBench-2, we changed how we build the metric. We decide how much each kind of error should matter first, then test and tune the scoring until it agrees. → 330 perturbation types to check that scores move by the right amount → 4,317 pages across six dimensions, now including forms → Full reference tables for all 1,025 charts → A Metric Lab to set scoring priorities for your own workload There’s still more to do, and we wanted to be open about that too. Read more about how we’re building toward a perfect document parsing benchmark in the reply below. 🔗 View Quoted Tweet📊 9 ⚡

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Products" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include LlamaIndex, ParseBench, document parsing, RAG.

Source attribution: curated from Jerry Liu (@jerryjliu0). Cross-source corroboration: 11 independent sources have covered this development (including TestingCatalog, Replit, kimmonismus), confirming industry-wide signal strength.