epfl-dlab/JSONSchemaBench

JSONSchemaBench

 

JSONSchemaBench is a benchmark of real-world JSON schemas designed to evaluate structured output generation for Large Language Models (LLMs). It contains approximately 10,000 JSON schemas, capturing diverse constraints and complexities.

import datasets
from datasets import load_dataset

def main():
    # Inspect the available subsets of the dataset
    all_subsets = datasets.get_dataset_config_names("epfl-dlab/JSONSchemaBench")
    print("Available subsets:", all_subsets)
    # Example output: ['Github_easy', 'Github_hard', 'Github_medium', 'Github_trivial', 'Github_ultra', 'Glaiveai2K', 'JsonSchemaStore', 'Kubernetes', 'Snowplow', 'WashingtonPost', 'default']

    # Access a specific subset of the dataset
    subset_name = "Github_easy"
    github_easy = load_dataset("epfl-dlab/JSONSchemaBench", subset_name)
    print(f"Loaded subset '{subset_name}':", github_easy)

    # Load the entire dataset as a whole
    entire_dataset = load_dataset("epfl-dlab/JSONSchemaBench", "default")
    print("Loaded entire dataset:", entire_dataset)

if __name__ == "__main__":
    main()

Update (March 31st, 2025)

To improve inference efficiency and streamline data collation, we’ve decided to drop a small number of exceptionally long samples from the dataset.

We’re using the meta-llama/Llama-3.2-1B-instruct tokenizer, and the filtering criteria are as follows:

• —Github_easy: Samples longer than 1024 tokens — 5 out of 582 removed
• —Github_medium: Samples longer than 2048 tokens — 7 out of 593 removed
• —Github_hard: Samples longer than 8192 tokens — 4 out of 372 removed
• —Other subsets are not touched

Since the number of discarded samples is minimal, this change is expected to have at most a 1% impact on results.

⚠️ Important Update (March 10th, 2025)

We have restructured the dataset to include train/val/test splits. If you downloaded the dataset before this date, you might encounter errors like KeyError: 'Github_easy'.

To fix this issue, please follow one of the options below:

1. 1.Update How Subsets Are Accessed: If you previously used:

from datasets import load_dataset, concatenate_datasets, DatasetDict, Dataset

subset: DatasetDict = load_dataset("epfl-dlab/JSONSchemaBench")
subset["Github_easy"]

You can update it to:

from datasets import load_dataset, concatenate_datasets, DatasetDict, Dataset

subset: DatasetDict = load_dataset("epfl-dlab/JSONSchemaBench", name="Github_easy")
subset: Dataset = concatenate_datasets([subset["train"], subset["val"], subset["test"]])

1. 1.Load the Dataset in the Old Structure: If you need the previous structure, you can use a specific revision:

dataset = load_dataset("epfl-dlab/JSONSchemaBench", revision="e2ee5fdba65657c60d3a24b321172eb7141f8d73")

We apologize for the inconvenience and appreciate your understanding! 😊

📌 Dataset Overview

• —Purpose: Evaluate the efficiency and coverage of structured output generation.
• —Sources: GitHub, Kubernetes, API specifications, curated collections.
• —Schemas: Categorized based on complexity and domain.

📊 Dataset Breakdown

📥 Loading the Dataset

from datasets import load_dataset

dataset = load_dataset("epfl-dlab/JSONSchemaBench")
print(dataset)

🔍 Data Structure

Each dataset split contains:

• —"json_schema": The schema definition.
• —"unique_id": A unique identifier for the schema.

🚀 For more details, check out the [paper](https://arxiv.org/abs/2501.10868).

📚 Citation

@misc{geng2025jsonschemabench,
      title={Generating Structured Outputs from Language Models: Benchmark and Studies},
      author={Saibo Geng et al.},
      year={2025},
      eprint={2501.10868},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2501.10868}
}

License

This dataset is provided under the MIT License. Please ensure that you comply with the license terms when using or distributing this dataset.

Acknowledgements

We would like to thank the contributors and maintainers of the JSON schema projects and the open-source community for their invaluable work and support.