Vik Paruchuri explains why OCR benchmark scores mislead model choice
Flawed ground-truth answers, scoring scripts that reject correct output, and postprocessing tricks can shift rankings more than real capability differences. Engineers choosing between OCR models should run task-specific evals on their own documents instead of trusting vendor-cited aggregate scores. The talk's final chapter walks through building private tests with verified answers and consistent scoring.
Vik Paruchuri argues in a talk that public benchmark scores are unreliable for choosing OCR models. He cites errors in ParseBench, including a test expecting 310 gigawatts of solar-panel capacity when the correct value is 303, and 73% of its chart tests asking for values not printed on the chart. In olmOCR-bench, a model was penalized for correctly preserving a line break, and a 700-line postprocessor added 1 to 2 points, more than gaps between models. OmniDocBench uses edit distance, which penalizes a correct HTML table more heavily than a wrong number in Markdown. The talk also covers how to slice benchmarks by task and build private evals on your own documents.
Should you rely on public benchmark scores when choosing a model? @VikParuchuri argues "no" in this talk.
To illustrate this, he shows specific problems found in popular OCR benchmarks:
ParseBench:
• A test expects 310 gigawatts of solar-panel capacity when the correct value is 303. Additionally, 73% of its chart tests ask for values that aren't printed on the chart. (4:51)
• A test requires "Parish Priest:" to be bold even though those words aren't bold in the source document. (15:33)
• The scorer fails to recognize heading tags placed on separate lines. (16:15)
• dots.mocr ranks last among six models even though it beats the first-ranked model on content and tables. This is an example of how overall benchmark score can hide important differences in model capabilities. (14:48)
olmOCR-bench:
• A model correctly preserves a line break in a document and gets penalized for it. (10:30)
• A test expects "Republican Party" when a document says "Republican party". Copying the capitalization correctly makes the model fail. (10:49)
• A 700-line postprocessor that rewrites model output adds 1 to 2 points, more than many gaps between models. (12:12)
OmniDocBench:
• This benchmark uses edit distance, which penalizes a correct table written in HTML more heavily than a wrong number in Markdown. So the format matters more than the answer! (6:58)
This is just the tip of the iceberg, the video contains more examples.
Chapters:
0:00 Choosing a model when every vendor claims to be #1 1:54 How OCR benchmark scores improve without better OCR 5:30 Metric design and tests that reject correct answers 11:33 Postprocessing, saturation, and misleading model rankings 12:40 How to slice benchmarks by your task and check for bias 16:53 How to build fair tests with verified answers and consistent scoring 19:05 How to test OCR models on your own documents 24:10 Benchmark marketing, private evals, and how to choose a model
