augustinian-babylm/vpswap-checkpoint-scores

VP-Swap checkpoint scores

Per-item correctness on the VP-Swap benchmark for nine models at twenty points in training. This is the raw material behind Figures 4-6 of Augustinian BabyLM (paper, code).

Layout

<model>/<revision>.jsonl, one line per benchmark item:

{"property": "color", "line": 6, "which": 1,
 "pll_orig": -21.4213, "pll_swap": -23.9077, "correct": true}

property and line identify the source line in eval/vpswap_bb24/vp_swap_<property>_pairs.txt; which is 1 or 2, since each line yields two items (each sentence is scored against the other's noun). Join to vp_swap_<property>_pairs.meta.jsonl by line number for corpus frequency, seeded status, and syntactic frame.

pll_orig and pll_swap are MLM pseudo-log-likelihoods: mask each token in turn, sum the log-probability of the original. An item is correct when the compatible sentence scores higher.

Models and revisions

Nine models: deberta-base-75k and its -s2 / -s3 seed replicates, deberta-base-75k-sam and its replicates, and the three synthetic-extension models deberta-base-75k-sam_ext-s1/s2/s3.

Twenty revisions each: step0, chck_1M through chck_10M, then chck_20M through chck_100M. Revision names match the branches on the corresponding model repos, so scores and weights can be lined up directly.

Why this might be useful

Minimal-pair benchmarks are usually reported as a single accuracy at the end of training. Here every item is scored at twenty checkpoints across three random seeds, with per-item metadata, so you can ask when an effect appears, which words carry it, whether it is stable across seeds, and how it relates to corpus frequency — without retraining anything.

step0 scores sit at chance (0.49-0.51) in every model, which is the unbiasedness check for the probe.

License

CC BY 4.0. Please cite the paper above.