Datalab Releases OmniParseBench OCR Benchmark

Datalab has launched OmniParseBench, an open evaluation harness built to audit document OCR parsers using deterministic pass or fail unit tests. Running on Python 3.11 or higher, the Apache 2.0-licensed code scores models offline without network calls or API keys. The dataset contains 16,288 tests across 2,937 pages and 2,343 documents in 92 languages. English content accounts for 10,089 tests, while non-English material spans 1,931 tests across 529 pages and 321 documents. Additionally, 6,767 table tests cover 1,000 pages from 875 documents.

Unlike alternatives such as Ai2's olmOCR-bench (7,010 tests across 1,403 PDFs), LlamaIndex's ParseBench (169,011 rules across 2,078 pages), or OpenDataLab's OmniDocBench (1,651 PDF pages), OmniParseBench eliminates subjective model judges. Overall headline scores average three core family scores: text, tables, and layout. Five test types—present, repeat, table_cell, and layout_kind—evaluate digit-for-digit text accuracy and KaTeX-rendered equations. Tests are categorized across structure, rendering, role, script (ISO 15924), and language (ISO 639-3) axes, tracking 10 distinct derivation sources.

Initial testing places Gemini 3.8 Flash at the top with a 92.9 headline score (91.3 text, 95.2 tables, 92.2 layout), followed by Datalab accurate at 92.7 (90.8 text, 91.8 tables, 95.6 layout) and Datalab balanced at 91.6 (90.0 text, 89.2 tables, 95.5 layout). Reducto scored 90.8 with a leading 95.8 layout score, followed by LlamaParse at 90.4 (88.2 text, 92.6 tables, 90.4 layout), Mistral OCR at 89.5 (82.8 text, 91.9 tables, 93.8 layout), and Claude Sonnet 5.5 at 88.7 (93.1 text, 95.4 tables, 77.5 layout). Rounding out the results were GPT-5.6 Sol at 87.8 (85.4 text, 91.6 tables, 86.5 layout), Extend at 83.3 (64.1 text, 92.5 tables, 93.4 layout), Azure at 83.0 (74.6 text, 81.7 tables, 92.8 layout), and Tesseract with 17.6 in text and 0.0 in tables.

For machine learning engineers, the framework translates broad leaderboards into actionable granular data by allowing sliceable evaluation by tag, alongside latency and cost metrics calculated from 200 sampled pages sent eight at a time. Developers can install the tool using uv pip install omni-parse-bench to inspect specific parser failure modes before deployment.