LlamaIndex launches ParseBench-2 benchmark and OCRia-1.5 parsing models
The severity-weighted metric design is the notable shift: it fixes the known weaknesses of both deterministic scores and LLM judges, making cross-vendor OCR comparisons more trustworthy. Engineering teams building RAG or document pipelines should re-run their parsing stack against ParseBench-2 before procurement decisions. OCRia-1.5 is available through LlamaParse in the cloud, giving developers a direct baseline to test against incumbent OCR providers on tables, forms and complex layouts.
LlamaIndex has released ParseBench-2, a document parsing benchmark with 4,300+ document pages, 330+ error types each assigned a severity score, and a new evaluation dimension covering forms. The company paired the benchmark launch with OCRia-1.5, a new family of document parsing models that powers all modes in its LlamaParse product. ParseBench-1, the predecessor, has reached 50,000+ monthly downloads with over 150 OCR solutions benchmarked and adoption by NVIDIA, Cohere, Aleph Alpha and Perplexity.
Unlike existing benchmarks that rely on brittle deterministic metrics (such as OmniDocBench) or LLM-as-a-judge approaches (such as ParseArena) that are coarse, stochastic and prompt-biased, ParseBench-2 tunes each metric against the severity scores so results align closely with human judgment. The ground truth across all dimensions is substantially improved over ParseBench-1, and page count has doubled.
OCRia-1.5 claims state-of-the-art performance on complex tables, forms, charts, formatting and bounding-box assignment for complex layouts. Teams evaluating OCR and document-parsing pipelines can use the benchmark's granular error taxonomy to compare vendors on the error types that matter most to their workflows, via the technical report at llamaindex.ai and the parsebench.ai site.
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with human judgment. We do this by painstakingly assigning severity scores to 330+ error types, and tuning each metric to match the severity scores. Existing benchmarks either use brittle deterministic metrics (e.g. OmniDocBench) or LLM-as-a-judge (ParseArena), which can help recognize equivalent representations but are coarse/stochastic/subject to bias by the prompt+model. ParseBench-2 also contains: ✅ 4.3k+ document pages (double from ParseBench-1). ✅ An extra dimension (forms) ✅ Much better ground-truth across all dimensions compared to ParseBench-1. This is the next evolution of ParseBench, our hugely popular benchmark that has 50k+ monthly downloads, over 150+ OCR solutions benchmarked, and has been adopted by companies like NVIDIA, Cohere, Aleph Alpha, Perplexity, and others. Alongside ParseBench-2, we are also releasing OCRia-1.5 - a new family of document parsing models that powers all our modes in LlamaParse. They represent a massive step up in capabilities from our latest release, and are state of the art in parsing complex tables, forms, charts, formatting, and assigning bounding boxes to complex layouts. Come check out our blog post, technical report, and website. Blog: llamaindex.ai/blog/introduci… Technical Report: llamaindex.ai/files/parseben… Website: parsebench.ai If you’re interested in trying out our OCRia-1.5 capabilities, come sign up for LlamaParse: cloud.llamaindex.ai
This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include LlamaIndex, ParseBench, OCRia-1.5, LlamaParse.
Source attribution: curated from Jerry Liu (@jerryjliu0).
