Cross-Mergeability/crossarch-accuracy

crossarch-accuracy — downstream accuracy for the ~1B merge results

Companion to `Mergeability-2/crossarch-1b-diagnostics`, which measured 106 Pythia-1.4B / Zh-Pythia-1.4B checkpoint-merge pairs, 1 native cross-model merge and 20 transport merges entirely in nats/token. This repo adds the accuracy axis: does any of it produce a usable model?

Headline question (Figure 1). The diagnostics found that checkpoint pairs within about half a decade of training are linearly mode-connected (median LMC barrier −0.033 nats) while pairs beyond 1.5 decades are not (+0.380). If Δaccuracy of the merge against the better parent crosses zero at the same separation, the barrier — computable from step numbers plus one cheap NLL sweep — is a free predictor of merge usability.

Benchmarks

Direct loglikelihood scorers, lm-eval-harness formats, validated against published EleutherAI/pythia-1.4b numbers:

ACC in the tables is the macro-average of the four tasks the 1.4B parents are above chance on (ARC-Easy, SciQ, PIQA, LAMBADA); its pooled chance level is 0.25. Belebele is reported but is at chance at this scale and must not be read as signal.

Files

• —results/table1_checkpoint_accuracy.csv — Table 1, one row per checkpoint pair
• —results/acc_ledger.jsonl — raw per-model per-task accuracies (resumable ledger)
• —results/summary.json — crossover estimates and correlations with permutation nulls
• —figures/fig1_crossover_*.png — Figure 1
• —figures/fig2_cka_vs_gain.png — Figure 2
• —scripts/ — the harness, verbatim

Work in progress; published incrementally as cells land.