dvader13/forecastgen-artifacts

1---2license: apache-2.03task_categories:4  - question-answering5  - text-generation6tags:7  - reasoning8  - evaluation9  - math10  - benchmark-results11  - model-outputs12size_categories:13  - 100K<n<1M14---15 16# Forecast-Generalization: raw evaluation outputs across 38 reasoning models17 18Complete generation-level outputs, per-seed scores and analysis artifacts from a19study of how well benchmark performance **forecasts generalization** to held-out20reasoning tasks.21 22Most released evaluations report only aggregate accuracy. This release keeps the23**raw per-problem, per-seed generations**, so item-level analyses can be redone24without re-running any inference.25 26## What is here27 28**38 models** evaluated on **four competition-style reasoning benchmarks**, each29run with **five random seeds** (42, 123, 456, 789, 1024).30 31| benchmark | evaluation runs |32|---|---|33| HMMT | 36 |34| GPQA | 30 |35| AIME | 30 |36| IMO | 22 |37 38Model families covered include OLMo-3 (7B/32B, Think and Think-SFT variants),39DeepSeek-R1 distillations (Llama-8B/70B, Qwen-7B/14B/32B), Qwen3, Gemma-4,40Seed-OSS-36B, Kimi, MiniMax and others — spanning roughly 7B to 70B parameters41plus several API-served models.42 43## Layout44 45| path | contents |46|---|---|47| `outputs/` | one directory per (benchmark, model, run): per-seed `*_predictions_seed-N.tsv` and `*_logs_seed-N.txt` |48| `expert_outputs/` | reference/expert model generations, grouped by `benchmark_model` |49| `scores/` | per-model accuracy CSVs: `name, seed_42 … seed_1024, mean_accuracy, std_error, num_seeds` |50| `data/` | benchmark inputs |51| `proxy_results/*.tar.gz` | proxy-metric evaluation results (next-token-prediction losses, sweeps) |52| `plots.tar.gz` | generated figures |53| `MANIFEST.tsv` | per-archive file counts, byte sizes and sha256 |54 55## Why the tarballs56 57`proxy_results/` and `plots/` together hold about 129,000 small JSON and image58files. Uploaded raw they would exceed practical per-repository file counts, so59they are stored as gzipped tars split along their natural subset boundaries.60 61Every archive was verified with `tar -d` (a member-by-member diff against the62source filesystem) after creation, and `MANIFEST.tsv` records the source file63count, archive member count, size and sha256 for each. Extraction restores the64original tree exactly:65 66```bash67tar -xzf proxy_results/ntp_results.tar.gz68```69 70## Example: seed variance on AIME71 72```python73import pandas as pd74from huggingface_hub import hf_hub_download75 76p = hf_hub_download("dvader13/forecastgen-artifacts",77                    "scores/aime_scores_seeds.csv", repo_type="dataset")78df = pd.read_csv(p)79print(df.sort_values("mean_accuracy", ascending=False)80        [["name", "mean_accuracy", "std_error"]].head())81```82 83Each row is one model; the five `seed_*` columns give per-seed accuracy, so84seed-level variance is directly measurable rather than inferred.85 86## Intended uses87 88- Item-level analysis (which problems a model solves, not just how many)89- Seed-variance and evaluation-reliability studies90- Correlating cheap proxy metrics against downstream benchmark accuracy91- Cross-model comparison on identical problems and identical seeds92 93## Notes94 95- Generations are raw model output, unfiltered, and include incorrect answers —96  that is deliberate, since error analysis is a main intended use.97- Benchmark problem statements belong to their original sources and retain the98  licenses of those benchmarks; the license above applies to the outputs and99  analysis artifacts collected here.100