weichy2023/FutureWorldv2

Offline Forecasting Benchmark (v5)

2067 resolved forecasting questions settling between 2026-04-01 and 2026-08-29. Every row has a ground truth. Answers are known, so this is an offline benchmark: a system is given an information cutoff and must not use evidence published after it.

This dataset is updated. Question count, taxonomy and per-class counts change between releases. Pin the revision you evaluated on and report it beside any score; a number without a revision is not comparable. See the changelog at the bottom.

Composition

Sources

Questions come from prediction markets, public data series and a third-party forecasting benchmark. The source field is preserved as-is so provenance stays auditable; rows converted from another benchmark are not relabelled as the market they resemble.

Subject taxonomy

Three levels. l0 is the kind of system that produces the outcome, l1 the observable domain, l2 the topic. The vocabulary is closed: l1 is exhaustive and every l2 belongs to exactly one l1.

Classes with zero rows are kept in taxonomy_v3.json as declared coverage gaps.

How the taxonomy was built

Earlier releases labelled bottom-up with industry tags, which mis-assigned systematically: "who becomes a trillionaire first" sat under single-stock price, "Elon leaves DOGE" under crypto (DOGE is a government office, not the coin), and Hong Kong rainfall under politics. v3.1 replaces this with a top-down scheme — fix an exhaustive closed schema first, then classify into it — which cut the others bucket from 83 rows to 15.

Names come from academic structure rather than being invented. The three l0 system domains line up with a university's faculty split (natural & medical / humanities, social & business / information & engineering): discipline boundaries are themselves cut by "what mechanism produces the state", which is the split we want. l1 uses department vocabulary; l2 names follow university department listings.

Labelling is choose, not create: taxonomy_v3.json is the closed schema, and annotation is a two-stage forced choice (one of 13 l1, then one l2 within that domain) enforced at the decode layer, not asked for in a prompt. others stays selectable on purpose — forbidding it pushes unplaceable questions into the nearest class and pollutes it. Ten case-law rules pin the boundaries that otherwise get labelled two ways (price vs physical, attention vs subject, a regulator's decision vs the regulated domain, and so on). Method references: arXiv:2601.12259, arXiv:2508.11987.

A single annotator labelled the current release and there is no inter-annotator agreement yet, so treat the taxonomy as a first pass.

Scoring

Discrete rows (BINARY, MULTIPLE_CHOICE) expect one option verbatim plus a probability for that answer. NUMERIC rows expect a finite real number.

`gt_history` is not shipped. It holds the resolved series behind each numeric question, so publishing it beside the question would give away the answer. The consequence, stated rather than hidden: the paper's Eq. 13 needs sigma(V) from that history, so without it numeric rows can only be scored by absolute error.

numeric_trivial

210 of 666 numeric rows (32%) are marked numeric_trivial: copying the last observation before the cutoff already scores above 0.95 under Eq. 13. They are kept because they still separate models, but a report that does not break them out will overstate numeric ability. The remaining 456 rows are the discriminating subset.

Known limits

• —Questions are English-language and skew toward US and European subjects.
• —eco and soc are thin; they are real coverage gaps, not a labelling artefact.
• —Rows converted from BTF-3 were binned into options here. The bin edges come from BTF-3's cutpoints, not from an original market, and are documented as such.
• —Market-sourced rows settle by their own market's rules, which are not uniform.

Changelog