epiq-ai-labs/corporatebench

CorporateBench

Dataset release for CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases.

CorporateBench evaluates information extraction, retrieval, and question answering over four synthetic corporate corpora ranging from 353 to 232,692 released documents. The corpora are generated from temporally evolving knowledge bases, providing deterministic ground truth across related documents.

Dataset Viewer

<https://corporatebench.epiqai.com/>

Quickstart

Setup:

uv sync --extra openai --extra anthropic

Run QA:

import asyncio

from corporatebench import CorporateBench, OpenAIBackend


async def main():
    async with CorporateBench(company="zenith") as bench:
        result = await bench.run(
            OpenAIBackend("gpt-5-nano"),
            method="rag",
            limit=10,
        )
        print(result.score)


if __name__ == "__main__":
    asyncio.run(main())

Run knowledge base evaluation:

import asyncio

from corporatebench import CorporateBench, OpenAIBackend


async def main():
    async with CorporateBench(company="zenith") as bench:
        result = await bench.run_kb_evaluation(
            OpenAIBackend("gpt-5-nano"),
            max_requests=bench.kb_evaluation_requests,
        )
        print(result.entities.macro_f1)


if __name__ == "__main__":
    asyncio.run(main())

Run topic classification for Biocure or Pound:

The overrides below run 31 requests instead of the 15,000 requests used by the full defaults.

import asyncio

from corporatebench import CorporateBench, OpenAIBackend


async def main():
    model = OpenAIBackend("gpt-5-nano", max_output_tokens=500)
    async with CorporateBench(company="biocure") as bench:
        result = await bench.run_topic_classification(
            model,
            shots=(0,),
            repetitions=1,
            test_size=31,
            max_requests=31,
        )
        print(result.scores_by_shots)


if __name__ == "__main__":
    asyncio.run(main())

What is this?

LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.

Dataset structure

data/
├── kb/                 # One .kb archive per company
├── kb_qa/              # Knowledge-base QA questions
├── topic_qa/           # Document-topic QA questions
├── integrated_qa/      # Questions combining graph and topic evidence
└── additional_qa/      # Supplemental topic-hierarchy configurations

Each .kb file is a ZIP-compatible archive containing:

Company configurations

Tasks and evaluation

The three QA tasks contain 3,000 core questions. The paper evaluates them with either document retrieval (RAG, k=10) or SQL access to the ground-truth KB, with at most five tool calls per question. Scalar comparison is case-insensitive; list answers receive partial credit through set F1.

QA schema

Every core QA file is a JSON object with metadata and questions fields.

The supplemental files under additional_qa/ use the same schema and provide alternate topic-hierarchy levels. Topic-classification examples and labels are read from the documents and topic graph inside the biocure and pound KB archives.

Construction and validation

1. 1.Define an RDF ontology for companies, departments, teams, employees, projects, tasks, meetings, and temporal relationships.
2. 2.Generate a valid organizational hierarchy, then simulate tasks, meetings, and hierarchy changes over a 90-day quarter.
3. 3.Annotate entities and serialize the world as an ontology-validated N-Quads graph.
4. 4.Generate corporate documents around graph-derived evidence strings. Email structure is modeled on the Enron corpus; topics, employee personalities, and formality levels provide variation.
5. 5.Derive QA answers through manually verified SPARQL queries rather than model generation.

Responsible use

CorporateBench is intended for research and evaluation of retrieval, extraction, and reasoning systems. It contains synthetic organizations and communications, not real corporate records. Generated names may nevertheless coincide with real people by chance.

Cite us

@inproceedings{hamilton2026corporatebench,
  title = {{CorporateBench}: Large-Scale {Q\&A} Benchmarking with Temporal Knowledge Bases},
  author = {Hamilton, Sil and Sun, Albert Yu and Romero, Oscar J. and Henneking, Carl-Leander and Mimno, David and Yang, Bishan and Labutov, Igor},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  publisher = {Association for Computational Linguistics},
  address = {Budapest, Hungary},
  month = {October},
  year = {2026}
}