Cross-Mergeability/crossarch-1b-diagnostics

Cross-architecture mergeability diagnostics for five ~1B monolingual LMs

A third model family for the mergeability project, alongside Goldfish and Beetle/MergeBench.

Read this first: what "merging" means for these five models

Five independently trained ~1B monolingual models were requested: Pythia-1.4B (EN), Zh-Pythia-1.4B (ZH), Tucano-1b1 (PT), Bielik-1.5B-v3 (PL), Minerva-1B (IT). They differ in architecture family, hidden dimension (1536 vs 2048), depth (16 / 22 / 24 / 32), head count (12 / 16 / 32), and vocabulary (32 000 / 32 768 / 49 953 / 50 304).

The naive weight-space operators -- averaging, task arithmetic, TIES, DARE, SLERP -- are undefined for 9 of the 10 pairs. They all compute elementwise functions of aligned parameter tensors, which presupposes a shared parameter space. Of the 10 pairs, exactly one has one natively: Pythia-1.4B x Zh-Pythia-1.4B (both GPT-NeoX, 2048 / 24 / 16), and even there only for the transformer blocks -- Zh-Pythia was retokenised to a 49 953-entry vocabulary, so embed_in and embed_out have different row counts and are excluded.

Nothing in this report zero-pads or truncates a hidden dimension to force a weight comparison. Where a weight-space column is blank for a pair, it is blank because the quantity does not exist, and the coverage table (§2) says which difference blocks it.

But "undefined" applies to the naive operators, not to merging in general. Cui et al., Transport and Merge: Cross-Architecture Merging for Large Language Models (arXiv:2602.05495), align activations to infer cross-neuron correspondences between heterogeneous models, solve an entropic-OT transport plan over neurons and a second OT over layers, and fuse donor weights into the target's coordinate system through those plans. Rectangular plans handle 1536 vs 2048 natively; the layer-level OT handles 16 vs 32. That is a principled change of basis, not padding, and §7 reports what it does here, together with a random-plan control: the identical fusion with the transport plans' rows shuffled, which destroys the learned neuron correspondence while preserving every marginal and the scale. Without that control a null result is indistinguishable from a broken reimplementation.

So there are three routes in this report, and a reader must not confuse them:

Post-transport weight quantities are not the same object as native weight-space quantities: they are measured after a fitted change of coordinates, so they carry the transport's error and its hyperparameters. Every such column is labelled transport_*.

The single biggest lever on statistical power

The 10 cross-model pairs are too few to correlate diagnostics against each other. So the primary arm of this report is within-run checkpoint merging: pairs of checkpoints from a single training run share architecture, vocabulary and parameter space exactly, so the full weight-space family is defined with no caveats, and the trajectory distance between two checkpoints is a natural continuous predictor. That arm supplies the n the correlation table needs.

Method

Parallel data. FLORES-200 (openlanguagedata/flores_plus), devtest split, all 1012 sentences, index-aligned across engLatn / cmnHans / porLatn / polLatn / ita_Latn. Each model sees its own language's side, tokenised by its own tokenizer; the only thing the two sides of a pair share is the sentence index. That is what makes the comparison well posed across different vocabularies.

Representations. SGPT position-weighted mean pooling over non-pad tokens of hidden_states[-1] -- the final pre-logit hidden state, post final layer-norm. Depths differ (16/22/24/32), so per-layer trajectories are not comparable and only the final pre-logit layer is reported, as the paper does.

Matched vs shuffled control. Side A in order; side B reordered by a single fixed-seed permutation drawn once per pair and reused for every metric of that pair.

Metrics. Linear CKA (Kornblith et al. feature-space form), orthogonal-Procrustes alignability (semi-orthogonal R of shape (dB, dA), so rectangular pairs are native), SVCCA (99% variance, capped at 64 components), and retrieval P@1 after fitting a ridge map on a 50% train split and retrieving on the held-out half by cosine. CKA carries a delete-a-group jackknife CI (50 blocks). All linear algebra runs in float64 on GPU -- this box's CPU is saturated.

Typology. URIEL via lang2vec, cosine distance on syntax_knn (syntactic), fam (genetic), and syntax_knn+phonology_knn+inventory_knn (featural).

Quality. Mean per-token NLL on the model's own FLORES side.

0. Headline findings

1. 1.Representational alignment is far above the shuffled control for all 10 cross-model pairs. Matched last-layer CKA spans 0.488-0.800 (median 0.587) against a shuffled control of 0.024-0.069. Retrieval P@1 after a fitted linear map reaches 0.949 against a chance rate of 0.0020. Five different architectures in five different languages encode FLORES sentences in near-affinely-related spaces.

1. 1.That alignment does not make the models weight-mergeable. The one pair with a genuinely shared parameter space (Pythia-EN x Zh-Pythia) has a weight cosine of 0.0163 over its shared blocks -- they are effectively orthogonal, having been trained from scratch with no common ancestor. Linear interpolation of their blocks raises EN NLL from 3.31 to 10.48 nats/token at the midpoint, a barrier of 6.22. High CKA and zero weight-space compatibility coexist.

1. 1.In the native arm, trajectory distance dominates every diagnostic. Over 106 within-run checkpoint pairs (78 Pythia + 28 Zh-Pythia), log10(stepb/stepa) correlates with the task-vector cosine at rho=-0.91, with TIES sign agreement at -0.88, with CKA at -0.85 and with the best merge gain at -0.67. The pattern replicates independently in both families.

1. 1.Nearby checkpoints are linearly mode-connected; distant ones are not. For pairs within half a decade of each other (n=24) the median LMC barrier is -0.033 nats/token -- at or below zero, i.e. the straight path between them is no worse than its endpoints. Beyond 1.5 decades (n=39) the median barrier is +0.380. Merging nonetheless almost never beats the better endpoint (6% of all 106 pairs), which is expected: within one run the later checkpoint is simply the better model, so there is no complementary knowledge to combine. Checkpoint merging is the right instrument for measuring the geometry, not for producing a better model.

1. 1.The diagnostics are highly redundant. Weight cosine and relative L2 correlate at rho=-0.98; task-vector cosine and sign agreement at 0.99. A mergeability panel of a dozen columns is measuring far fewer than a dozen things.

1. 1.Transport-based cross-architecture fusion is measurable but, without adaptation, not useful. All 20 directed pairs fused successfully, including every rectangular case (1536 <-> 2048 hidden, 16 <-> 32 layers) -- the OT plans are rectangular by construction, so nothing was padded. The learned neuron correspondence is doing real work: the OT plan beats its own shuffled-plan control on the target's language in 12/20 directions. But fusion alone improves the target in only 2/20 directions; the median effect is +0.0021 nats/token. The exception is instructive: the best result is PL_bielik <- ZH_pythia at -0.0534, and the two directions that help at all both have Bielik-PL as the target -- the weakest model in the set (NLL 3.77). That is exactly the paper's setting: transfer INTO a low-resource target. Cui et al. recover their headline gains in the post-fusion adaptation step, which we did not run.

1. The five models, as actually configured

Config check. Four of the five match the brief. Bielik-1.5B-v3 does not: its config reports model_type: llama with 32 layers, not a 28-layer Qwen2.5. Its geometry (hidden 1536, 12 heads, 2 KV heads, FFN 8960) is Qwen2.5-1.5B's, so it is a Qwen2.5-derived model re-exported under the Llama class with a 32k Polish-adapted vocabulary. Everything below uses the true config, not the brief's.

Access note. speakleash/Bielik-1.5B-v3 and sapienzanlp/Minerva-1B-base-v1.0 are auto-gated on the Hub. The gates were accepted programmatically with the project token to obtain the weights; the Bielik gate is a checkbox agreeing to be contacted for feedback. Flagging it because it was accepted on the account owner's behalf.

2. Coverage: which pairs were merged, by which route, and why

3. Table 3 equivalent -- last-layer matched vs shuffled CKA

Lower triangle = matched. Upper triangle, italic = shuffled control.

4. The 10-pair cross-model diagnostic matrix

Per-model quality (mean per-token NLL on that model's own FLORES-200 devtest side):

URIEL source: lang2vec/URIEL (cosine distance on syntax_knn / fam / syntax+phon+inv).

5. Checkpoint merging -- the native weight-space arm

All of these are native merges: same run, same architecture, same vocabulary, same parameter space. No alignment, no transport, no padding.

By trajectory distance:

4b. This family beside the project's other two

The one number that is comparable across the three studies is last-layer, SGPT-pooled, FLORES matched CKA. Read the note column before comparing rows -- the Goldfish values were transcribed upstream and were not recomputed under this protocol.

6. The one natively weight-mergeable cross-model pair: Pythia-EN x Zh-Pythia

Shared parameter space covers 290 tensors / 1,208,602,624 parameters (every transformer block). Excluded because the shapes differ: embed_out.weight, gpt_neox.embed_in.weight -- Zh-Pythia's 49 953-entry vocabulary against Pythia's 50 304.

Weight cosine over the shared blocks: 0.0163; relative L2 1.577; norm ratio 0.328.

Only base-free operators are defined here (the two models have no shared pre-trained ancestor, so task vectors theta - theta_base are meaningless across them). t=0 is the host alone; t=1 replaces every block with the donor's.

7. Transport-based cross-architecture merging

Directional: the target keeps its architecture, tokenizer, embeddings and LM head. Donor weights are carried into the target's coordinate system by entropic-OT neuron plans plus a layer-level OT, then fused on a top-128 neuron mask. Fusion only -- no post-fusion adaptation, which is where Cui et al. get most of their gain. Deviations from the released code are listed in the header of scripts/transport_merge.py; the load-bearing one is that we rescale each plan to be row-stochastic, without which the transported term is numerically ~0.

Deviations from the paper / the released code, all deliberate and all reproducible from scripts/transport_merge.py:

• —Plan normalisation. Neither the paper nor the released code rescales the OT plan Q, whose rows sum to 1/n. The transported term is then smaller than the donor weight by ~1/(n_out n_in), so the fusion degenerates into a (1-alpha) shrinkage of the masked entries rather than a change of basis. We use the standard OT-fusion convention Phi = n * Q so each plan is row-stochastic.
• —Epsilon 0.1 -> 0.01. Our neuron costs 1-rho have mean 1.00 and sd 0.07-0.10; at the paper's eps=0.1 the plan sits close to the uniform coupling (row max 0.006 against a uniform 0.0005) and fusion through it is indistinguishable from fusion through a shuffled plan -- which is exactly what the random-plan control showed. At eps=0.01 the plan is a soft near-assignment whose transport cost is within 1% of the unregularised optimum. Evidence: results/ot_epsilon_calibration.txt.
• —Sublayer coverage. The released code transports attention q/k/v/o. Across these five models the attention sublayer is not commensurable (GPT-NeoX fuses QKV; head counts 12/16/32; GQA with 2 or 4 KV heads), so we transport the interfaces every one of them shares: the MLP (d_model -> d_ffn -> d_model) and the attention output projection. q/k/v keep the target's own values.
• —Layer-pair pruning and layer-cost subsampling. The outer OT needs only the scalar <C,Q> per layer pair, so that pass runs on a fixed random subsample of at most 2048 neurons per side; fusion uses full dimensions, on the top-3 donor layers per target layer, with P_eff renormalised over the kept entries.
• —No post-fusion adaptation.

Negative delta_* means fusion reduced NLL (helped); positive means it hurt. nll_donor_lang is the target model evaluated on the donor's language with the target's own tokenizer -- the setting where transferred donor knowledge would show up. ot_minus_rand_* is the control: negative means the OT plan beat a shuffled plan.

The control splits the pairs cleanly, and the split is interpretable. Every direction with Bielik-PL as target has the OT plan beating its shuffled control on both languages -- transporting into the weakest model in the set is where the learned correspondence pays. Every direction with Zh-Pythia as donor has the OT plan doing markedly worse than random. That is not the plan being uninformative -- an uninformative plan scores zero on this axis, as the middle of the ranking does -- it is the plan being confidently wrong. The probe sides there are Chinese against a Latin-script language, and a neuron correspondence fitted on 512 mean-pooled samples across that gap appears to concentrate transported mass on the wrong neurons. More probe data (the paper uses 2000) is the obvious first thing to try.

8. Diagnostic x diagnostic Spearman correlations -- checkpoint arm (native weight merging)

n is adequate here. Permutation null: 20000 label shuffles per cell; q_bh is Benjamini-Hochberg across the whole table.

8. Diagnostic x diagnostic Spearman correlations -- cross-model arm

n=10. This is badly underpowered. Spearman rho on 10 points has a 95% CI roughly +/-0.6 wide; nothing short of |rho|>0.65 clears an uncorrected permutation test, and nothing at all survives BH correction across 55 comparisons. Read the signs, not the significance. Permutation null: 20000 label shuffles per cell; q_bh is Benjamini-Hochberg across the whole table.

9. Figures

[image]

matched vs shuffled last-layer CKA, 5 models

[image]

diagnostic correlation heatmap, checkpoint arm

[image]

diagnostic correlation heatmap, cross-model arm (n=10)

[image]

checkpoint merge results vs trajectory distance

[image]

transport-based cross-architecture merge

What was cut, and why

The brief allowed 5 hours and instructed scoping down over overrunning. Cut, explicitly:

• —Per-layer CKA trajectories. Depths differ, so they are not comparable; the paper reports the final pre-logit layer and so does this.
• —Downstream task evaluation of merged models. Merge quality is scored by NLL on held-out FLORES only. No BLiMP / MMLU-style benchmark was run: it would have consumed the entire window for one column.
• —Post-fusion supervised adaptation for the transport merges. Cui et al. report "fused w/o adaptation" and "fused w/ adaptation" separately and get most of their gain from the adaptation step. Only the no-adaptation arm is reported here, which is the conservative arm and the one that isolates what the transport itself buys.
• —A second probe set. Everything uses FLORES-200 devtest. No Tatoeba / BouQUET replication.
• —Seed replication. One permutation seed per pair, one DARE seed. No seed-level error bars on the merge operators.
• —Two further checkpoint families. TucanoBR/Tucano-1b1 publishes 23 stepN revisions and sapienzanlp/Minerva-1B-base-v1.0 publishes 30. Both would have extended the native arm to four families. Cut for disk and wall clock; the two Pythia families already replicate each other, and the checkpoint blobs were deleted after measurement to return ~157 GB to a shared volume.
• —Attention q/k/v in the transport merge, and the transport probe set at 512 rather than the paper's 2000 samples, and an alpha grid of only {0.01, 0.03, 0.1}. See the deviation list in section 7.
• —A tuned alpha per pair. Cui et al. select alpha per task on the evaluation benchmark. We report the whole grid instead, which is the more conservative presentation.

Provenance

Code: /root/crossarch-diagnostics/scripts/. Metric implementations are self-contained (metrics.py, float64 on GPU) and were cross-checked pairwise against the mergeability project's Goldfish-faithful mergeschool.geometry.gf_core.linear_cka: maximum absolute disagreement 1.0e-15 across all 10 pairs (results/estimator_crosscheck.json). /root/mergeability was read, never written. Raw per-pair records are in results/*.csv and results/ckpt_*.jsonl; the OT epsilon calibration evidence is in results/ot_epsilon_calibration.txt.

After measurement, the 22 Pythia / Zh-Pythia checkpoint revisions were deleted from the shared HF cache (~157 GB returned); the main revision of each of the five models was kept.