Cross-Mergeability/extrinsic-evaluations

Extrinsic evaluations — the union view

One tidy long-format table of every extrinsic (downstream, task-level) evaluation produced across the 2026-08-26 mergeability workstreams, so that a single file answers "how did model X score on benchmark Y" regardless of which experiment produced it.

The per-experiment datasets remain the authoritative record of their own methods, figures and caveats. This is the union view, not a replacement, and it deliberately carries no analysis of its own.

• —extrinsic_evaluations.csv / .parquet — 9,675 rows, one per measurement
• —build_manifest.json — per-source row counts, fetch status, dedupe and conflict counts
• —build_extrinsic.py — the build script; re-runnable and idempotent

Read this before you read a number

The one thing this file exists to keep straight

A likelihood rescue and an accuracy rescue are not the same thing, and tonight established that they are uncorrelated. A file that silently mixed nats/token with benchmark accuracy in one value column would be actively misleading, so:

• —metric always names its unit explicitly (nats_per_token, accuracy_pct, …).
• —metric_kind is the coarse grouping you should always group by before aggregating: accuracy · accuracy_delta · benchmark_score · likelihood · likelihood_delta · other.
• —unit spells the unit out again.

Never aggregate across `metric_kind`. Never treat one as a proxy for the other. The evidence for that instruction is in this file — see Supported claims below.

Every accuracy row has a chance level

chance is populated for all 2,215 metric_kind == "accuracy" rows, in the same units as value. It is empty for likelihood rows (where chance is meaningless) and for benchmark_score rows, which are suite aggregates over heterogeneous tasks — MergeBench's per-domain scores, and the mean_of_benchmarks rows — where no single chance level is defined. That is why those rows are labelled benchmark_score and not accuracy.

Schema

Metric dictionary

Nats per UTF-8 byte appears wherever two parents use different tokenizers, because nats/token is not comparable across them. This is the source's choice, preserved here.

Provenance

Not yet included

Still being produced when this file was built. build_extrinsic.py registers it and skips it cleanly; re-running once it publishes folds it in with no code change.

Mergeability-2/beetle-merge-eval is included, but it was still being written during the build — its long table grew from 419 to 493 rows while this file was being assembled. Treat its 492 rows as partial coverage and re-run the builder for the complete set. Its coverage_by_merge.csv records which merges are complete, partial or not evaluated.

How the rows were deduplicated

Several sources are resumable ledgers that re-report shared baselines: the SET 1 set1 / set1x / slerp / repair ledgers all re-emit identical M0_naive_avg and M1_perm_avg rows for the same merges, and set1x_410m re-measures three pairs already in set1_410m. Left alone that would have triple-counted the baseline rungs and inflated the 410m cell from 15 pairs to 18. The build assigns each merge a canonical identity and drops repeats: 3,485 repeated measurements removed. Per-size counts now reproduce the source's own rung_summary.csv exactly (410m: n = 15, mean Δfloor 6.5290 / 5.9558 / 6.0254).

60 residual conflicts are recorded in build_manifest.json. All are BLiMP accuracies differing between two ledgers by exactly one item out of 6,700–13,400 — evaluation nondeterminism, not disagreement. The dedicated BLiMP ledger wins.

Verification

Every extractor was checked against its source's own published tables before release:

• —PolyPythia 14m Δfloor by rung — 32.4294 / 9.6065 / 16.0073 / 146.7951 / 241.5204 — exact match
• —PolyPythia BLiMP by rung and size (14m 0.518 / 0.533 / 0.530 … 410m 0.535 / 0.543) — exact match
• —AIM T0_published_outcome_paired spot values — exact match
• —crossarch checkpoint pair counts (Pythia 78, Zh-Pythia 28) — exact match
• —chat-vector aligned − naive on every real fork — exactly 0.000 on every benchmark, as claimed

One inconsistency in a source is worth flagging: compose-audit's headline §4 quotes the pythia-14m aligned merge at BLiMP 0.544, while its own authoritative table reports 0.533. This file carries 0.533, the table value. It does not change the direction of the finding.

What this data supports, and what it does not

Supported

The likelihood rescue does not transfer to accuracy. On PolyPythia seed pairs — same data, same architecture, same tokenizer, so the merge obstruction is purely coordinate — permutation alignment removes ~70% of the naive merge's Δfloor at 14m, and the merged model still scores 0.533 on BLiMP against parents at 0.652 and chance 0.500. Across the ladder the merged model sits between 0.518 and 0.543 at every size, whether alignment recovered three quarters of the likelihood gap or a tenth of it. Pair by pair the two rescues are uncorrelated (Spearman 0.14 / −0.17 / 0.17 / 0.05 / 0.23). Both arms are in this file, on the same merges, in separate metric_kind groups — which is the whole reason the file is shaped this way.

The dissociation runs in both directions. Goldfish merges whose Δfloor says they are destroyed still score 0.68 on MultiBLiMP-English (parent 0.96, chance 0.50). Neither metric implies the other.

Alignment's coordinate rescue decays with scale. The exactly function-preserving permutation rung removes 70% of the naive Δfloor at 14m and 8% at 410m. The coordinate-removable share of the obstruction is falling in the direction the field is scaling. (The source reports this as a mean of per-pair percentages; recomputing it as a ratio of means from these rows gives 70% → 9%. Same conclusion, slightly different estimator — read the source's tables for the canonical figure.)

Naive averaging of two seed-only-different LMs is catastrophic at every size, and alignment does not make it usable: even the best rung leaves the merge at or above the uniform-over-vocabulary reference at the small sizes.

Joint training beats every merge tested. The B-GPT jointly_trained rows are the ceiling, and no merge in this file reaches them on either metric.

AIM's published benefit is real. Paired across 20 matched checkpoints, the with-AIM arm is up on the endpoint-scaled benchmark mean (18/20 positive, Wilcoxon p = 9.5e-06). Those published outcomes are here for both arms.

The headline nulls — read these before quoting anything positive

0 of 1035 MergeBench cells survive Benjamini-Hochberg at n = 8 families (80 pairs). No property-vs-outcome relationship in the MergeBench panel is distinguishable from zero at the coverage its published outcomes permit. An earlier draft reported 306 surviving cells; all were artefacts of three defects since fixed — a cluster bootstrap with a 40.2% false-positive rate at n = 4, a saturation gate calibrated on pair-level rather than family-level spread, and testing against zero when the families cluster by architecture. This dataset carries only MergeBench's published outcome scores, not those correlation cells — the cells live in Mergeability-2/mergebench-property-sweep, and table_domain_cells.csv is what to read.

0 of 13 released Llama-3.1-8B derivatives have left the base parameterisation. Across every released derivative examined — language forks, domain continued pretraining, instruct post-training, a safety model — not one had moved out of the base model's coordinate frame. The aligned and naive chat vectors are therefore bit-identical models, and the accuracy difference in this file is exactly 0.000 on every benchmark, for every fork, at every λ. That is a null about the ecosystem, not about the mechanism: the control rows, where a real fork is acted on by a random element of its own symmetry group, show the naive chat vector collapsing (IFEval 0.175 → 0.110, below the fork it started from) and alignment restoring it to 0.355. The mechanism reaches accuracy. The condition that would make it pay off did not occur in any released model.

Pre-merge predictors do not reliably predict the realised rescue — 0 of 25 cells significant, held out by seed pair. The strongest predictor does not replicate across substrates (held-out AUROC 0.48 / 0.71 / 0.81 / 0.61 / 0.45).

AIM does not change what makes a merge work. Not one of 216 property-vs-outcome correlation cells survives multiplicity correction, and the with-minus-without change is inside the noise band from re-splitting the same benchmarks within one arm.

Not supported — do not use this file for these

• —Pair-level MergeBench merge outcomes. They do not exist. Every score MergeBench publishes is a five-expert merge (merge_arity = 5 in all 576 rows), so the 80-pair property suite has no published counterpart. parents says so on every MergeBench row.
• —Cross-source comparisons of raw `value`. Substrates, corpora, item budgets and scorers differ between experiments. Compare within an experiment, or within a (experiment, benchmark, metric) group.
• —Anything about merged-model accuracy at 1B+ scale. The 1B arm in this file is likelihood-only; the accuracy arm (Mergeability-2/crossarch-accuracy) was not published at build time. The checkpoint-distance rule — keep souped checkpoints within about half a decade of training steps — is measured in nats/token, not benchmark accuracy, and given the dissociation above it must not be assumed to transfer.
• —Complete Beetle coverage. 492 rows are in, but the source was mid-run: of the merges it tracks, several are still not evaluated. Read coverage_by_merge.csv in the source before quoting a Beetle aggregate.
• —A practitioner diagnostic. The scientific claim (merge failure can arise from how a function is represented, and that is measurable) is well supported. The practitioner claim (compute this before merging and it tells you what to do) is not supported at the coverage available.
• —Significance testing. This file carries measurements, not tests. Every p-value, bootstrap and permutation null lives in the source datasets.

Rebuilding

source /root/.ms_hf_env      # HF_TOKEN
python3 build_extrinsic.py            # rebuild from the Hub
python3 build_extrinsic.py --push     # rebuild and publish

Idempotent: it re-fetches each source, skips any that 404, rebuilds every row from scratch, re-runs the dedupe, and refuses to emit an accuracy row without a chance level. Re-run it when crossarch-accuracy publishes, and again once beetle-merge-eval finishes its run.

Row counts

3,472 distinct models. By role: 8,698 merged · 323 parent · 248 jointly_trained · 210 control · 196 reference.

Example

import pandas as pd
df = pd.read_parquet("extrinsic_evaluations.parquet")

# the dissociation, in two lines
seed = df[df.experiment == "polypythia_seed_merge"]
acc  = seed[(seed.metric == "accuracy") & (seed.model_role == "merged")]
nats = seed[(seed.metric == "delta_floor_nats_per_token") & (seed.model_role == "merged")]
print(acc.groupby(["provenance_family", "rung"]).value.mean())   # flat, ~0.52-0.54
print(nats.groupby(["provenance_family", "rung"]).value.mean())  # collapses with alignment

# never do this
df.groupby("benchmark").value.mean()   # mixes nats with accuracy. group by metric_kind.