Datasetpublic

Cross-Mergeability/beetle-merge-eval

Beetle merged models — benchmark evaluation against their parents Minimal-pair benchmark accuracy for the Beetle merged models published in the Mergeability org, scored against their own parent models and, where one exists, the jointly-trained ceiling on the same harness. The existing sweep datasets (Mergeability/merge-sweep-results, Mergeability-2/mergeability-results) record merge quality in nats (NLL, delta_floor, rel_damage, barrier, geometry). They contain no downstream… See the full description on the dataset page: https://huggingface.co/datasets/Cross-Mergeability/beetle-merge-eval.

sourceHugging Faceupdated 27d agoView on Hugging Face

0likes305downloads

Dataset Card

Beetle merged models — benchmark evaluation against their parents

Minimal-pair benchmark accuracy for the Beetle merged models published in the `Mergeability` org, scored against their own parent models and, where one exists, the jointly-trained ceiling on the same harness.

The existing sweep datasets (Mergeability/merge-sweep-results, Mergeability-2/mergeability-results) record merge quality in nats (NLL, delta_floor, rel_damage, barrier, geometry). They contain no downstream benchmark accuracy for these repos — this dataset adds that layer.

Files

file	what it is
`beetle_merge_eval_long.csv`	tidy long format: `merge_id, provenance_family, parent_a, parent_b, ceiling, rung, operator, arm, benchmark, metric, value, chance, n, eval_langs, kind` — every table regenerates from this
`beetle_merge_eval_deltas.csv`	the same rows plus `naive_value`, `parent_a_value`, `parent_b_value`, `better_parent`, `ceiling_value`, `d_naive`, `d_parent`, `d_ceiling`, `z_vs_chance`, `at_chance`, `near_chance`
`coverage_by_merge.csv`	per-repo: rungs published vs rungs scored, and status
`summary_*.csv`	per-family / per-operator aggregates
`RESULTS_BEETLE_MERGE_EVAL.md`	what ran, what did not, the honest verdict per family
`fig_headline_naive_vs_aligned.png`	naive-vs-aligned and better-parent-vs-merged scatters, faceted by provenance family
`fig_merge_vs_joint_ceiling.png`	merged vs jointly-trained counterpart, for the two families that publish one
`arm_identity_audit.csv`	per operator-block: which arms exist, and whether aligned/transport are byte-identical to naive
`harness_validation_vs_beetle_analyze.csv`	cell-by-cell agreement with beetle-analyze published numbers
`retest_reliability.csv`	the 1,026 cells independently scored twice by two waves — a repeat-measurement check on the harness

kind == "reference" rows are the parent and ceiling models themselves; kind == "merge" rows are the merged rungs. Chance is 0.5 for every minimal-pair benchmark and 50 for BLiSS.

Benchmarks

English BLiMP (nyu-mll/blimp, 67,000 pairs), MultiBLiMP (jumelet/multiblimp, per language), BLiMP-NL (juletxara/blimp-nl, 9,000), ZhoBLiMP (Junrui1202/zhoblimp, 35,400), and BLiSS (ALTACambridge/BLiSS) where present. MultiBLiMP has no `zho`, `jpn`, `kor` or `fil` config, so Chinese pairs get ZhoBLiMP instead.

Harness

Scoring is a frozen vendor of `beetle-analyze` (src/beetle_analyze/eval/eval_blimp.py, eval_bliss.py): sum log-probability per sentence, accuracy = fraction where logp(good) > logp(bad); BLiSS metrics are a direct port of compute_bliss_metrics. The only change is that the fp32 [B,T,V] log-softmax is evaluated in row chunks (identical values, bounded memory), and rung subfolders are staged to disk because trust_remote_code does not honour subfolder= when resolving auto_map.

Caveat that dominates everything else

For every Beetle pair checked, the published __aligned folder holds byte-identical weights to its __naive sibling (verified on LFS sha256). Δ vs naive is therefore exactly 0 for aligned rungs, by construction, and no conclusion about weight alignment can be drawn from these artefacts. Only __transport differs from __naive.

Scope

Minimal-pair benchmarks only. BLiSS was not run — the scorer is vendored and working and a 79-model job set is prepared, but the pass was dropped on the user's instruction, not for any technical reason. A small pilot is retained under benchmark == "bliss" and labelled as such. MECO L2 ΔlogL and JFLEG were also not run; RESULTS_BEETLE_MERGE_EVAL.md carries the exact commands for all three.