Datasetpublic

brianward92/draftfm-frozen-eval

DraftFM frozen_eval: the per-pick prediction archive Every prediction behind the tables in the DraftFM paper, archived so the results can be recomputed (no GPU needed) and audited pick by pick. Model: brianward92/draftfm Code and paper: https://github.com/brianward92/mtga Each top-level directory is named by the sha256 of the frozen data snapshot it was evaluated against — the directory name itself is the registration. Inside are per-pick prediction parquets for every model in… See the full description on the dataset page: https://huggingface.co/datasets/brianward92/draftfm-frozen-eval.

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes254downloads

Dataset Card

DraftFM frozen_eval: the per-pick prediction archive

Every prediction behind the tables in the DraftFM paper, archived so the results can be recomputed (no GPU needed) and audited pick by pick.

—Model: brianward92/draftfm
—Code and paper: https://github.com/brianward92/mtga

Each top-level directory is named by the sha256 of the frozen data snapshot it was evaluated against — the directory name itself is the registration. Inside are per-pick prediction parquets for every model in the battery (scaling rungs s1–s16, f-dev, f-full, ablations, and baselines), in both human and deployment scoring modes, plus derived accuracy curves and a summary.json recording the evaluation context and gates.

Directories

Directory	What it is
`013df16b…183a`	The headline MSH pass: single pre-registered zero-shot evaluation over the first public MSH snapshot (PremierDraft, protocol v1.1, executed 2026-07-07). Source of the 57.0% top-1 result
`013df16b…183a_postday1`	Post-day-one MSH ceiling: per-set model fine-tuned on MSH after the zero-shot pass, as pre-registered in the protocol (2026-07-09)
`df6220d8…853`	Full-battery end-to-end rehearsal on SOS PremierDraft (protocol v1, 2026-07-05), run before the real MSH pass to burn in the pipeline
`6552b298…34e`	Earlier SOS TradDraft rehearsal with baselines (protocol v1, 2026-07-04)
`validation/`	Per-set validation parquets (e.g. SOS) used for sanity gates

The rehearsals are included deliberately: the protocol's credibility comes from showing the pipeline was exercised and frozen before the one-shot MSH evaluation, not tuned after it.

Using it

Each <model>.<mode>.parquet holds one row per pick with the model's probability over the pack and the chosen card; summary.json in each directory records the snapshot sha256, S3 ETag, protocol tag, execution timestamp, and data-quality gates. To recompute a table, aggregate the parquets; to audit a claim, join any pick back to its prediction. The battery definition lives in the model repo's frozen_battery.json; the chronology is in the code repo's docs/eval_protocol.md.

Provenance and licensing

These files are model outputs and derived aggregates generated by the DraftFM pipeline (Apache-2.0). The underlying human draft data is the 17Lands public dataset (CC BY 4.0 — "Data from 17Lands.com"); raw 17Lands rows are not redistributed here. Unofficial Fan Content per the Wizards of the Coast Fan Content Policy; not approved or endorsed by Wizards.

Contact

Brian Ward — brian.ward.92@gmail.com