Datasetpublic

dvader13/forecastgen-artifacts

Forecast-Generalization: raw evaluation outputs across 38 reasoning models Complete generation-level outputs, per-seed scores and analysis artifacts from a study of how well benchmark performance forecasts generalization to held-out reasoning tasks. Most released evaluations report only aggregate accuracy. This release keeps the raw per-problem, per-seed generations, so item-level analyses can be redone without re-running any inference. What is here 38 models… See the full description on the dataset page: https://huggingface.co/datasets/dvader13/forecastgen-artifacts.

sourceHugging Faceapache-2.0updated 27d agoView on Hugging Face

0likes193downloads

Dataset Card

Forecast-Generalization: raw evaluation outputs across 38 reasoning models

Complete generation-level outputs, per-seed scores and analysis artifacts from a study of how well benchmark performance forecasts generalization to held-out reasoning tasks.

Most released evaluations report only aggregate accuracy. This release keeps the raw per-problem, per-seed generations, so item-level analyses can be redone without re-running any inference.

What is here

38 models evaluated on four competition-style reasoning benchmarks, each run with five random seeds (42, 123, 456, 789, 1024).

benchmark	evaluation runs
HMMT	36
GPQA	30
AIME	30
IMO	22

Model families covered include OLMo-3 (7B/32B, Think and Think-SFT variants), DeepSeek-R1 distillations (Llama-8B/70B, Qwen-7B/14B/32B), Qwen3, Gemma-4, Seed-OSS-36B, Kimi, MiniMax and others — spanning roughly 7B to 70B parameters plus several API-served models.

Layout

path	contents
`outputs/`	one directory per (benchmark, model, run): per-seed `_predictions_seed-N.tsv` and `_logs_seed-N.txt`
`expert_outputs/`	reference/expert model generations, grouped by `benchmark_model`
`scores/`	per-model accuracy CSVs: `name, seed_42 … seed_1024, mean_accuracy, std_error, num_seeds`
`data/`	benchmark inputs
`proxy_results/*.tar.gz`	proxy-metric evaluation results (next-token-prediction losses, sweeps)
`plots.tar.gz`	generated figures
`MANIFEST.tsv`	per-archive file counts, byte sizes and sha256

Why the tarballs

proxy_results/ and plots/ together hold about 129,000 small JSON and image files. Uploaded raw they would exceed practical per-repository file counts, so they are stored as gzipped tars split along their natural subset boundaries.

Every archive was verified with tar -d (a member-by-member diff against the source filesystem) after creation, and MANIFEST.tsv records the source file count, archive member count, size and sha256 for each. Extraction restores the original tree exactly:

bash

tar -xzf proxy_results/ntp_results.tar.gz

Example: seed variance on AIME

python

import pandas as pd
from huggingface_hub import hf_hub_download

p = hf_hub_download("dvader13/forecastgen-artifacts",
                    "scores/aime_scores_seeds.csv", repo_type="dataset")
df = pd.read_csv(p)
print(df.sort_values("mean_accuracy", ascending=False)
        [["name", "mean_accuracy", "std_error"]].head())

Each row is one model; the five seed_* columns give per-seed accuracy, so seed-level variance is directly measurable rather than inferred.

Intended uses

—Item-level analysis (which problems a model solves, not just how many)
—Seed-variance and evaluation-reliability studies
—Correlating cheap proxy metrics against downstream benchmark accuracy
—Cross-model comparison on identical problems and identical seeds

Notes

—Generations are raw model output, unfiltered, and include incorrect answers — that is deliberate, since error analysis is a main intended use.
—Benchmark problem statements belong to their original sources and retain the licenses of those benchmarks; the license above applies to the outputs and analysis artifacts collected here.