Datasetpublic

realzL/benchability-fig4-source-uniform

BenchAbility Figure 4 -- source_uniform One of two training mixtures drawn from the same frozen 884,143-row candidate pool, with the same budget (60,000 intervention + 15,000 shared replay) and the same hyperparameters. The two differ only in how the samples are chosen, which is the whole experiment. arm source_uniform selection by source provenance only, chart:doc:ocr = 3:4:4 intervention rows 60,048 replay rows 15,000 shards 38 pool 884,143 rows / 20… See the full description on the dataset page: https://huggingface.co/datasets/realzL/benchability-fig4-source-uniform.

sourceHugging Faceotherupdated 1mo agoView on Hugging Face

0likes192downloads

Dataset Card

BenchAbility Figure 4 -- `source_uniform`

One of two training mixtures drawn from the same frozen 884,143-row candidate pool, with the same budget (60,000 intervention + 15,000 shared replay) and the same hyperparameters. The two differ only in how the samples are chosen, which is the whole experiment.


arm	`source_uniform`
selection	by source provenance only, chart:doc:ocr = 3:4:4
intervention rows	60,048
replay rows	15,000
shards	38
pool	884,143 rows / 20 source datasets, capability-tagged sample-by-sample

Columns

column	meaning
`uid`	`source-split-index`, stable across both arms
`source`	original dataset (provenance)
`capability`	BenchAbility leaf, assigned per sample by a vision-language classifier
`split_role`	`intervention` or `replay`
`question` / `answer`	the training turn; `<image>` marks where the image goes
`image`	PNG bytes, embedded

capability is present in both arms so the mixtures can be compared, but the source_uniform draw never read it -- see below.

How this arm was drawn

Sources are grouped into the three coarse families Figure 2 reports (chart, doc, ocr) and drawn 3:4:4. Within a family the quota is split across sources proportional to sqrt(rows), then water-filled -- not equally, because equal shares would need FUNSD (149 rows) roughly 11 times over while the chart family never repeated a row, and unequal repetition between the arms would confound the comparison.

This arm never reads a capability label. The draw is handed rows with the field stripped. The arm exists to model an engineer who has only benchmark-level reporting; letting it see sample-level labels would make it a weaker copy of the other arm rather than the alternative it represents. Labels are attached afterwards, for auditing what the draw happened to contain.

Reproducing

bash

python fig4_training/pipeline/40_mix.py      # both arms from the frozen pool
python fig4_training/pipeline/50_export.py   # this bundle

Full draw record, including every relaxed constraint and every shortfall, is in mixture_manifest.json.