realzL/benchability-fig4-capability-guided
BenchAbility Figure 4 -- capability_guided One of two training mixtures drawn from the same frozen 884,143-row candidate pool, with the same budget (60,000 intervention + 15,000 shared replay) and the same hyperparameters. The two differ only in how the samples are chosen, which is the whole experiment. arm capability_guided selection by capability, gap-weighted from the Figure 3 scores intervention rows 59,999 replay rows 15,000 shards 38 pool 884,143… See the full description on the dataset page: https://huggingface.co/datasets/realzL/benchability-fig4-capability-guided.
BenchAbility Figure 4 -- capability_guided
One of two training mixtures drawn from the same frozen 884,143-row candidate pool, with the same budget (60,000 intervention + 15,000 shared replay) and the same hyperparameters. The two differ only in how the samples are chosen, which is the whole experiment.
Columns
capability is present in both arms so the mixtures can be compared, but the source_uniform draw never read it -- see below.
How this arm was drawn
Per-capability quotas come from the measured Figure 3 scores:
gap(l) = max(0, 95 - score(l)) weight(l) = 0.20/9 + 0.80 * gap(l) / sum(gap)
The scores are measured on the diagnosis benchmarks through scripts/build_fig1.py's own code path, not the placeholder table in the plan document. The target is 95 rather than 70 because the real scores top out at 93.69 -- at 70, seven of the nine capabilities have gap 0 and the experiment collapses into "train two capabilities".
Within each capability bucket, samples are drawn across benchmarks under three diversity bounds: a 50% per-source cap, a 15% per-template cap, and at most 2 rows per image. Where a bound is arithmetically unsatisfiable it is relaxed one at a time and the relaxation is recorded -- key_information_extraction is three templates in total by construction, and AI2D is 71% of diagram_and_infographic_understanding.
Two capabilities cannot be filled from this pool: diagram (7,264 available against 11,081 requested) and KIE (1,416 against 1,864). The deficit is redistributed over the capabilities that have room, in gap proportion. It is never filled by upsampling with replacement, which would give those buckets more effective epochs than the same rows get in the other arm.
Reproducing
python fig4_training/pipeline/40_mix.py # both arms from the frozen pool
python fig4_training/pipeline/50_export.py # this bundleFull draw record, including every relaxed constraint and every shortfall, is in mixture_manifest.json.