Solshine/lfm2.5-2.6b-nla-L21-av-priordev-relabel-v2

LFM2.5-2.6B NLA Activation Verbalizer (priordev-relabel-v2)

LoRA adapter for LiquidAI/LFM2.5-2.6B that takes a 2048-dimensional residual-stream activation captured at layer 21 and emits a short natural-language string.

Read this before you download it

This adapter does not produce usable descriptions of the activations you give it. It is a methodology artifact, released so that a negative cross-architecture result is checkable. It is not a working interpretability tool and should not be used as one.

What the measurements support, stated exactly: the adapter's output carries a statistically significant pool-wide retrieval signal, which this program's own routing-versus-content decomposition attributes principally to coarse topic routing rather than to reading the specific content of an activation. The within-domain axis, which is the one that isolates content, is not significant ($p=0.087$). Exact-document top-1 is at chance. And the generated text itself, read by a human, is confabulated: see the verbatim samples below.

An earlier version of this card called this "the strongest content-reading result in this program outside Gemma". That wording conflated routing with content, which is precisely the error the accompanying paper's decomposition exists to prevent, and it is withdrawn.

This is a cross-architecture port of the recipe developed on Gemma-4-E2B (`gemma-4-e2b-nla-L23-av-priordev-relabel-v1-wd3`), and it carries the larger pool-wide signal of the two ports, including against the same recipe applied to a larger model (Qwen3.5-4B).

What the outputs actually look like

Three consecutive rows from the held-out evaluation, by document id, with nothing selected for or against. The eval pool is deliberately out-of-domain: this adapter was trained on news, and these are dialogue and poetry.

The outputs are fluent, confident, shaped like the news corpus it was trained on, and unrelated to their inputs. They also leak training-template tokens (</concept>, </explanation>). The percentile effect below is measured against this, not against something more readable.

Why this is "v2": the previous attempt could not be measured at all

The immediately preceding attempt at this port failed its injection-sanity gate, so no retrieval number of any kind exists for it. That failure and its fix are the substance of this release.

Nothing else changed: same corpus, same hyperparameters, same 6200-step horizon. The v1 failure was not an injection-plumbing bug — the mechanism demonstrably perturbed generation — it was a diversity collapse, and swapping the objective's prior model eliminated it (0.157 → 0.939) rather than merely improving it.

The prior-deviation-weighted objective computes a per-token weight against what a "prior" model would predict. Gemma's winning recipe conditions that prior on a previously-trained verbalizer. Using an untrained base model instead is a weaker and differently-shaped prior, and on this architecture the difference was the difference between a measurable checkpoint and an unmeasurable one.

Evaluation

197-row held-out pool. Lower percentile is better (0.5 = chance); higher top-1 is better.

Real injection beats both controls on every metric, and the full-pool percentile sits 11.11 points off chance while both controls sit within noise of exactly chance (0.4982 and 0.5000).

What that does and does not mean. Full-pool percentile ranks the true document against the whole 197-document pool, so getting the broad topic neighbourhood right is enough to move it. Same-domain percentile ranks the true document only against others from its own domain, so it is the axis that isolates reading the specific document. The first is significant, the second is not. Under this program's own routing-versus-content decomposition, that pattern is a routing signal with no demonstrated content signal underneath it, and it is the same pattern the accompanying paper reports as a failure mode for its non-winning checkpoints. Top-1 of 3/197 against a 2/197 shuffle control is one extra hit and should be read as chance.

Comparison with Qwen3.5-4B, on matched methodology

Both families were trained with the identical L3 recipe, the same corpus size, and evaluated the same way, so this is a like-for-like comparison of the architectures rather than of the recipes.

The smaller model carries the larger routing signal. That is the comparison this table supports. It is not a statement that either adapter reads content: the same-domain axis, the one that would show content, is not significant for either family, and neither adapter produces output a human can use. Qwen3.5-4B's only advantage is injection diversity, which measures whether the mechanism is live and varied, not whether anything correct is being read. Model size did not predict which family showed the larger effect. Absolute magnitudes across the two families are not strictly comparable (different base models, eval pools and n), so read this as directional.

Honest limits

• —Top-1 is small-N. 3/197 vs 2/197 vs 1/197; the 95% Wilson intervals are [0.52%, 4.38%], [0.28%, 3.63%] and [0.09%, 2.82%] respectively, and they overlap substantially. The top-1 ordering is consistent with the percentile metrics but is not independently significant. The percentile means carry the claims here.
• —The same-domain gap is not significant. The full-pool gap is: paired per document over the 197 rows, real beats shuffle at Wilcoxon $p=0.00028$ two-sided, and $p=0.00015$ under a 20,000-draw sign-flip permutation test. The same-domain gap does not reach significance ($p=0.087$ and $p=0.053$), so the harder within-neighbourhood discrimination is not distinguishable from the shuffle control at this $n$, and this card's claims rest on the full-pool metric alone. An earlier version of this card listed "no formal paired significance test" as a limitation and bolded the same-domain figure alongside the full-pool one; the test has since been run and the same-domain figure is unbolded accordingly.
• —Corpus is smaller and narrower than Gemma's: 600 rows / 300 documents, all news, against Gemma's 1356 rows / 975 documents over 8 domains.
• —Evaluation is deliberately out-of-domain (code, poetry, dialogue, reviews, finance against a news-trained corpus). This is the harder setup.
• —Single training seed.

Training configuration

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("LiquidAI/LFM2.5-2.6B", dtype="float16", device_map="auto")
model = PeftModel.from_pretrained(base, "Solshine/lfm2.5-2.6b-nla-L21-av-priordev-relabel-v2")

The activation is L2-normalised, rescaled to the injection scale, and overwrites the input embedding at the marker token position. The adapter alone will not reproduce these numbers without that injection mechanism; see inject_config.json. Run the shuffled-activation and no-injection controls before trusting any generation-based number — this project has hit silent injection failures three separate ways, and the v1 attempt described above is a fourth distinct failure mode that only the diversity gate caught.

What ships in this repo

Every figure and table in this card is derived from the files in eval/, so the claims here are checkable against their own artifacts rather than only reproducible in principle.

Citation

Part of the GPU-poor NLA program. Full methodology and controls are in the accompanying paper; source repository: `SolshineCode/deception-nanochat-sae-research`.