AI startup Datalab has introduced OmniExtractBench, an open-source benchmark suite designed to evaluate structured document extraction systems. The benchmark brings together 620 documents drawn from four existing benchmarks and uses a unified scorer to measure how accurately models fill a JSON schema. Datalab said the evaluation framework relies on six interpretable judgment types, aiming to make scoring easier to inspect and compare across systems. The company also argued that existing extraction benchmarks suffer from bias, opaque scoring methods, and limited document diversity. To address part of that workflow, OmniExtractBench uses the Hungarian algorithm for content-based row alignment. Datalab said the scorer is available through PyPI, while the dataset is hosted on Hugging Face under a CC BY 4.0 license. The development was cited by MarkTechPost.
AI startup Datalab has released OmniExtractBench, an open-source benchmark suite for evaluating the performance of structured document extraction systems.
620 documents collected from four existing benchmarks
OmniExtractBench combines 620 documents from four existing benchmarks. It uses a unified scorer and six interpretable judgment types to evaluate how accurately models populate a JSON schema.
Issues Datalab sees in current extraction benchmarks
Datalab said existing extraction benchmarks suffer from bias, opaque scoring mechanisms, and a narrow range of document types.
How the scorer and dataset are distributed
The benchmark uses the Hungarian algorithm for content-based row alignment. According to the release, the scorer can be installed through PyPI, while the data is available on Hugging Face under a CC BY 4.0 license.
The update was cited by MarkTechPost.
This article was originally published by Bit.Fan. For more cryptocurrency news and market insights, visit www.bit.fan. Disclaimer:
The market information, project data, and third-party content displayed on this platform are for industry information sharing only and do not constitute any form of investment advice or return commitment.
Cryptocurrency trading carries high risks. Users should fully assess their risk tolerance and make independent decisions. All profits, losses, and legal responsibilities are borne by the users themselves.
