Duruo/forecastbench-single_question

ForecastBench Single Questions

This dataset contains single-ID forecasting questions derived from the ForecastBench project. It includes two configurations:

• —forecastbench_single_questions_2024-12-08: Contains 429 forecasting questions with resolved real-world outcomes.
• —forecastbench_single_questions_human_2024-07-21: Contains 473 questions with resolved real-world outcomes, augmented with human forecast probabilities from public and superforecaster groups.

Dataset Structure

Each entry in the dataset is a JSON object with the following fields:

• —id: Unique identifier for the question.
• —question: The forecasting question text.
• —background: Additional context or background information.
• —source: Origin of the question (e.g., "manifold").
• —source_intro: A standardized prompt prefix shown to forecasters, e.g."We would like you to predict the outcome of a prediction market. A prediction market, in this context, is the aggregate of predictions submitted by users on the website Manifold. You’re going to predict the probability that the market will resolve as ‘Yes’."
• —answer: Binary outcome (0 or 1) representing the resolved real-world result.
• —human_public_forecast: (Optional) Forecast probability made by the public human group.
• —human_super_forecast: (Optional) Forecast probability made by the superforecaster (expert) human group.

Usage

To load a specific configuration using the Hugging Face datasets library:

from datasets import load_dataset

# Load the base dataset without human forecasts
dataset = load_dataset("Duruo/forecastbench-single_question", name="forecastbench_single_questions_2024-12-08")

# Load the dataset with human forecasts
dataset_with_human = load_dataset("Duruo/forecastbench-single_question", name="forecastbench_single_questions_human_2024-07-21")

License and Attribution

This dataset is released under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0).

It is based on the original ForecastBench dataset, which is also licensed under CC BY-SA 4.0.

If you use this dataset, please cite the original ForecastBench project and provide appropriate attribution.

Citation

@inproceedings{karger2025forecastbench,
      title={ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities},
      author={Ezra Karger and Houtan Bastani and Chen Yueh-Han and Zachary Jacobs and Danny Halawi and Fred Zhang and Philip E. Tetlock},
      year={2025},
      booktitle={International Conference on Learning Representations (ICLR)},
      url={https://iclr.cc/virtual/2025/poster/28507}
}

:contentReference[oaicite:21]{index=21}

---

:contentReference[oaicite:23]{index=23} :contentReference[oaicite:26]{index=26}:contentReference[oaicite:28]{index=28}
::contentReference[oaicite:29]{index=29}