caiotheodoro/plumb
Plumb Gold tasks and the held-out eval. The study is on the collection. Config n What benchmark 1000 Held-out eval, seed 777. Never in train. train_handseeded 223 Mix matched to the eval, including PASS. train_ornith 58 Ornith-1.5 proposals that passed the oracle. train_blended 281 Both of the above. Leakprobe vs benchmark: exact signature overlap 0. curriculum train n sw-recall precision exact hand-seeded 223 0.318 [0.290, 0.347] 0.308 [0.279… See the full description on the dataset page: https://huggingface.co/datasets/caiotheodoro/plumb.
Plumb
Gold tasks and the held-out eval. The study is on the collection.
Leakprobe vs benchmark: exact signature overlap 0.
The evals config has all 15 published rows (full size, equal-N@58 random and stratified, clean equal-N@18) with their CIs, so the tables in the post are reproducible without rerunning anything:
from datasets import load_dataset
evals = load_dataset("caiotheodoro/plumb", "evals", split="train")
[r for r in evals if r["group"] == "full_size"]Raw per-prediction dumps live at data/evals/eval-{handseeded,ornith,blended}.json and PATH_A_RESULTS.json. Those are artifacts, not configs; fetch them with hf_hub_download.
CIs: 10k bootstrap, seed 11. Adapters: `handseeded` · `ornith` · `blended`. N=18 mix/error and pow-* / leaked-anchor grows are not here.
from datasets import load_dataset
bench = load_dataset("caiotheodoro/plumb", "benchmark", split="train")Apache-2.0. Synthetic pay applications, not real contractor filings.