Solshine/nla-gemma4e2b-relabel-v1-eval

Gemma-4-E2B layer-23 evaluation set, relabeled (v1), with contamination flags

The 580-document evaluation pool on which every activation-verbalizer result in this project is scored, with each row's evaluation text rewritten from a topic summary to a feature-attribution label. The activations are byte-identical to the original evaluation set; only the text column changed, and the original text is preserved.

This pool is not disjoint from the training corpus. Read this before using it.

The pool was split from the training corpus on document identifier, and the two corpora use different identifier conventions (opaque hashes here, readable slugs in the corpus), so the split reported zero overlap. Checked on text (normalized exact match, a shared verbatim run of 20 or more words, or 8-gram containment above 0.30), 383 of the 580 documents (66.0%) are present verbatim in `nla-gemma4e2b-relabel-v1-corpus`, and in the original un-relabeled corpus. The companion file contamination_flags_580.json carries the per-document result of that check. Any retrieval or usability number computed on this pool must be reported split by that flag, with the candidate pool held whole so chance stays 1/580; the project's own earlier "273–314x held-out retrieval" figure for the relabeled adapter was computed on the whole pool and is withdrawn on this basis (on the 197 clean documents that adapter's retrieval is at chance). The lesson generalizes: verify held-out status on text, never on identifiers.

Why it exists

The original evaluation labels were short topic summaries (median 15 words, e.g. "Mesophilic bacteria's optimal 37 C growth and their role in fermenting cheese and yogurt"). A within-domain content test that swaps activations over near-interchangeable summaries is structurally handicapped, so the same 580 rows were relabeled against the corrected feature-attribution rubric and the released base verbalizer was rescored under both label conditions. Result: within-domain content moved from 0.529 (not significant) to 0.557 (p = 0.007 versus chance), but the paired test between the two label conditions on the same rows was not significant (McNemar p = 0.24). The honest reading is that the base verbalizer's content signal sits in a narrow band that crosses the significance line depending on label style; this file lets anyone rerun that comparison.

Files

powered_eval_heldout_relabeled.parquet (580 rows). Columns follow the training corpus (activation_vector, doc_id, domain, response = new label, old_label_verbatim = original label, source text up to the cut). Exact column names should be confirmed against the file at upload time; see the release checklist in the source repository.

contamination_flags_580.json: the per-document output of the text-level disjointness check described above (the source repository's experiments/v8_nla_local/train_eval_disjointness.py, run against the relabeled corpus on 2026-09-12), keyed by this file's doc_id. Join on it before reporting anything.

Quality

An independent gate on a random 30-row sample: 27 pass, 3 borderline, 0 fail, zero fabricated facts. Relabeled texts are longer than the originals (mean 20.6 versus 14.7 words including the wrapper; both capped at 48 target tokens in the scoring harness).

Use

Pair with the adapters under the Solshine namespace and the scoring code in SolshineCode/nla-gemma-4-e2b; the numbers reported for this set are verifiable without a GPU from SolshineCode/gpu-poor-nla. Do not train on this file, and do not describe results on it as held-out without the contamination split: it is the evaluation set for every published number in the project, and two-thirds of it is in the training corpus.

License

CC BY 4.0 for the labels and metadata; source-text excerpts retain their original licenses.