anicka/nla-qwen3-4b-universal-av-grpo

NLA Activation Verbalizer — Qwen3 4B, universal, GRPO-refined

The universal SFT verbalizer sharpened with AR-native GRPO: the reward is whether an independent Activation Reconstructor can rebuild the original activation from the description alone. No semantic similarity model anywhere in the reward — faithfulness is scored in the model's own 2560-dim activation space. Smallest model in the series to date (4B), and the one that gains the most from GRPO (+31% relative round-trip vs +24% for the 7B sibling).

Part of the nla-at-home project.

What GRPO changed (eyeball version)

SFT describes the genre of the activation; GRPO quotes its content. Same activation (math text, L18):

SFT: "False" as a negation marker active, with "True" as a competing attractor · "logarithmic" and "exponential" as a contrastive pair (round-trip cos 0.69) GRPO: False statement: "The statement is false" (direct negation of the claim) · Logical structure: the quadratic equation x² + 4x + 4 = 0 has a repeated root at x = −2, which is a real number (round-trip cos 0.90)

Reward design (the recipe that worked)

reward = centered_cos(AR(desc), act) × specificity(‖AR(desc) − μ_L‖)

• —centered cosine: per-layer mean removed from both sides — raw cosine is ~90% shared layer offset and rewards nothing.
• —specificity: a generic description reconstructs to a small-norm centered vector; multiplying by the reconstruction norm makes genericness unprofitable in activation space, with no contrastive hacks.
• —no hard negatives, no MiniLM in the gradient, KL anchor (β 0.01) to the base model without the adapter, repetition penalty 0.2 (the SFT stage's repetition loops were an explicit target).
• —curriculum, not reward: an oracle compass (per-layer ridge map activation→text embedding, fit fresh on this model's AR, val top-1 0.59–0.75) selects which (layer, text) pairs are decodable enough to train on, tau 0.40→0.10. The compass never touches the reward or the eval.

Trained on 9 of 36 layers (0, 4, 9, 13, 18, 22, 26, 31, 35), group size 6, 400 samples/epoch, lr 5e-6.

Two-stage history (full provenance ships with the repo)

The original 8-epoch run was interrupted mid-epoch-4 (best: epoch 3, reward 0.5302, tau at 0.27). Training resumed from that checkpoint as a fresh GRPO run with the tau schedule continuing 0.27→0.10 over 5 epochs; best checkpoint is continuation epoch 3 (reward 0.5330). The KL anchor is the adapter-free base model in both stages, so the two stages are regularization-compatible. Curriculum consumption logs for both stages ship as trained_samples_stage1.jsonl + trained_samples.jsonl, and the eval holdout below excludes texts touched by either stage.

Evaluation — round-trip on a clean holdout

284 texts that none of AV training, AR training, or either GRPO stage ever touched (eval_holdout_ids.json = the SFT validation split minus every text either curriculum consumed). Descriptions generated greedy, reconstructed by the frozen AR, centered cosine:

GRPO wins at every depth. The extreme layers (L0, L35) stay weak for both — a property of the 4B substrate, not the training stage.

The confabulation tail

Fraction of confidently-wrong descriptions (low round-trip cosine), all nine layers / content band only (L4–L31):

In the content band the sign-flipped tail (cos < 0) drops 10×. Raw per-item data: roundtrip_eval.records.jsonl.

Injection protocol

Identical to the SFT card: ㈎ (id 149705), normalize TO L2 norm 150.0, depth-conditioned prompt (nearest_depth_pct(L, 36)), chat template with enable_thinking=False. GRPO-sharpened adapters are numerically touchy: use CUDA (bf16/fp16) or CPU fp32 — not MPS bf16.

Limitations

Same corpus scope as the SFT stage (safety-filtered public split). GRPO improves content specificity; it does not add knowledge of activation regions the corpus never covered. Near-input (L0) and final-layer (L35) readouts remain unreliable at this model scale.