CorporateBench Q&A Benchmark Released
CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases
CorporateBench provides developers with a realistic benchmark for evaluating LLM performance on enterprise-scale document collections, addressing the lack of shared corporate data.
CorporateBench (CB) is a human-validated multi-task Q&A benchmark with evaluation corpora exceeding 230,000 documents.
It evaluates LLMs across information extraction and knowledge base querying dimensions using four synthetic firms ranging from 12 to 10,000 employees.
Five LLMs tested showed increasingly poor performance as input size approached realistic scales.CB fills a crucial gap in the benchmarking ecosystem for corporate communication reasoning.
CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases
LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include CorporateBench, LLM, knowledge bases, enterprise documents.
Source attribution: curated from arXiv cs.AI (@Sil Hamilton, Albert Yu Sun, Oscar J. Romero, Carl-Leander Henneking, David Mimno, Bishan Yang, Igor Labutov).
