allenai/omega-500

Omega-500: Random Sample of Mathematical Problems

This dataset contains a random sample of 500 mathematical problems selected from the comprehensive OMEGA problem families dataset. It provides a diverse, manageable subset for quick evaluation and experimentation across multiple mathematical domains and difficulty levels.

Overview

Omega-500 is designed for:

• —Quick Evaluation: Fast assessment of model capabilities across math domains
• —Prototyping: Testing new approaches before scaling to larger datasets
• —Benchmarking: Standardized subset for fair model comparisons
• —Research: Focused analysis on a balanced mathematical problem set

The sample maintains diversity across mathematical domains and difficulty levels while keeping the dataset size manageable for rapid iteration.

Quick Start

from datasets import load_dataset

# Load the Omega-500 sample
dataset = load_dataset("allenai/omega-500")
problems = dataset["train"]

# Access individual problems
first_problem = problems[0]
print("Problem:", first_problem["messages"][0]["content"])
print("Answer:", first_problem["ground_truth"])
print("Family:", first_problem["family"])
print("Difficulty:", first_problem["difficulty_level"])

Dataset Composition

Total Problems: 500

Domain Distribution:

• —Algebra: 92 problems (18.4%)
• —Arithmetic: 173 problems (34.6%)
• —Combinatorics: 84 problems (16.8%)
• —Geometry: 45 problems (9.0%)
• —Logic: 61 problems (12.2%)
• —Number Theory: 45 problems (9.0%)

Data Fields

Each problem contains:

• —id: Unique identifier for this sample
• —original_id: Original identifier from source dataset
• —family: Problem family (e.g., "algebrafuncarea")
• —difficulty_level: Numeric difficulty level from source
• —source_family: Source family directory name
• —source_level: Source difficulty level name
• —messages: Problem statement in chat format
• —ground_truth: Correct answer
• —dataset: Dataset identifier ("OMEGA500SAMPLE")

Citation

If you use this dataset, please cite the original OMEGA work:

@article{sun2024omega,
  title     = {OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization},
  author    = {Yiyou Sun and Shawn Hu and Georgia Zhou and Ken Zheng and Hannaneh Hajishirzi and Nouha Dziri and Dawn Song},
  journal   = {arXiv preprint arXiv:2506.18880},
  year      = {2024},
}

Related Resources

• —Full Problem Families: See omega-problems for the complete dataset
• —Explorative Dataset: See omega-explorative for explorative reasoning challenges
• —Compositional Dataset: See omega-compositional for compositional reasoning challenges
• —Transformative Dataset: See omega-transformative for transformative reasoning challenges
• —Paper: See the full details in paper
• —Code Repository: See generation code on github