MinghuiXu/countdown_fresh_heldout_1024

Countdown Fresh Held-Out 1024

This dataset contains 1,024 fresh held-out Countdown arithmetic problems for evaluating language models on the 3-to-4 number Countdown task.

Each problem provides a target integer and a list of 3 or 4 numbers. A model must construct an arithmetic expression using each number at most once and the basic operations +, -, *, and / to equal the target.

Files

• —standard/test.parquet: prompts for standard no-tool evaluation.
• —tool/test.parquet: prompts for tool-augmented evaluation.
• —summary.json: generation metadata and distribution/overlap checks.

Generation Procedure

The benchmark was generated locally with seed 20260530. The public dataset asingh15/countdown_tasks_3to4 was used only to estimate the training distribution and to build an exclusion set.

For each candidate problem, we:

1. 1.Sampled the target and number count to match the public training distribution.
2. 2.Sampled numbers in the range [1, 99].
3. 3.Rejected duplicates and any candidate whose (target, sorted(numbers)) key appeared in the public train or test splits.
4. 4.Retained only candidates with an exact arithmetic solution found by a deterministic solver.

The resulting set contains 502 three-number problems and 522 four-number problems. The target range is 10 to 100, with mean target 54.84. The overlap with public train/test by (target, sorted(numbers)) is zero.

Intended Use

This dataset is intended as a larger in-distribution held-out evaluation set for pass@k experiments. It is useful when the original public test split of 50 examples is too small for stable estimates.

This is not an external human-curated benchmark. It is a synthetic fresh held-out set designed to follow the same Countdown task distribution while avoiding known public train/test problem keys.

Citation

If you use this dataset, please describe it as a fresh held-out Countdown evaluation set and report the generation/filtering procedure above.