LlamaIndex releases ParseBench-2 forms dataset with 20K annotated fields
Severity-weighted scoring is the non-obvious idea here: the cost of a parsing error depends on the field type, so a lost signature and a misplaced decimal should not count equally. Teams building invoice, compliance, or government-form pipelines can rerun their OCR or model stack against parsebench.ai and adjust the severity weights to match their own error tolerance. The dataset's 30-country government forms also make it a useful stress test for document AI serving multilingual markets.
LlamaIndex has introduced a new forms-focused component within ParseBench-2, a benchmark for evaluating document parsing systems. The forms dataset contains 600 pages drawn from 244 government form templates spanning 30 countries, with roughly 20,000 annotated fields covering text entries, checkboxes, and signatures.
The dataset is designed around the specific challenges of form parsing: templates vary widely, and elements like checkboxes and signatures need different representations than plain text. A good parser must extract both empty and filled values and return exact bounding boxes for each field. LlamaIndex also proposes a severity-weighted evaluation metric — blank fields must stay blank, formatting changes cost nothing, a lost signature counts 5, and a wrong number costs a minimum of 10.
The team notes that form parsing remains difficult regardless of whether specialized OCR engines or frontier models are used, and argues no single severity policy fits all use cases, so practitioners are encouraged to tune their own weighting. The benchmark is available at parsebench.ai and the dataset is published on Hugging Face, with details in a LlamaIndex blog post and tech report.
Parsing forms is a hard problem, whether with specialized OCR or frontier models. There is no standardization across various form templates. There are multiple form elements (e.g. checkboxes, signatures, text fields) that need to be represented in various ways. A good parser needs to both extract empty/filled form values along with exact bounding boxes. We’ve spent time carefully crafting a forms dataset within ParseBench-2. It consists of 600 pages from 244 government form templates across 30 countries, with 20K annotated fields (text, checkboxes, signatures). We also designed a forms metric that’s weighted by severity policies, like the following: - Blank fields have to stay blank - Formatting changes cost nothing - A lost signature counts 5 - A wrong number costs min 10 Different form extraction use cases will prioritize different errors more, and we encourage you to tune your own severity policies for form parsing. More details in our blog / tech report: llamaindex.ai/blog/introduci… Benchmark: parsebench.ai Available on Hugging Face! huggingface.co/datasets/llama…
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include LlamaIndex, ParseBench, OCR, RAG.
Source attribution: curated from Jerry Liu (@jerryjliu0).