Tincan0325/smoea-15ood-benchmark

SMoEA fixed 15-OOD benchmark data

The evaluation data for scripts/run_rejection_benchmark.py. Not needed for interactive or batch mode — only the benchmark reads it.

Getting it

scripts/setup_workspace.sh downloads this for you into dataset/ood_data/, which is what --benchmark-root defaults to. To fetch it on its own:

python scripts/fetch_benchmark.py --repo Tincan0325/smoea-15ood-benchmark

That verifies the per-group counts on arrival. The plain hf command works too, but does no checking:

hf download Tincan0325/smoea-15ood-benchmark --repo-type dataset \
    --local-dir dataset/ood_data

Layout — one flat directory

dataset/ood_data/
├── ni_task_descriptions.json
├── task149_test.json    task476_test.json    task933_test.json
├── task1622_test.json   task1670_test.json                       1722 instances
├── bbh_test.json                                                 1187 instances
└── mmlu_pro_test.json                                            1250 instances

Same shape as dataset/train_data/, so both live under dataset/.

Earlier revisions of this repository also carried the three-level MoEA directory tree (prompts/, ood/data/, dataset/natural_instructions/data/selected_10_tasks/test_data/). The files were byte-identical to the flat ones above. system/benchmark.py still reads that layout if it finds it, so an existing local copy keeps working.

Usage

python scripts/run_rejection_benchmark.py \
    --artifact ties_only \
    --output-dir results/rejection-ties_only

Add --smoke to run one instance per family as a pipeline check. Without it, the loader enforces the exact counts below and refuses to start if any differ. --artifact takes any id declared in the registry, including base, arrow and the four taskwise variants.

Contents

Every instance is an answer-free full prompt: {instance_id, full_prompt, output}, with definition at the top level of each NI file. BBH and MMLU-Pro instance ids carry the dataset in the second :: field (bbh::causal_judgement::00000) — that is how the loader splits them apart.

Checksums

From the directory you downloaded into:

sha256sum -c SHA256SUMS

Sources

Preprocessed from BIG-Bench-Hard, MMLU-Pro and Natural Instructions: prompts were rendered from the task templates, sampled to the fixed counts above, and stripped of trailing answers. Redistribution follows each upstream dataset's own licence.