cds-jb/gemma4-26b-a4b-ao-l20-v3

gemma4-26b-a4b-ao-l20-v3 — Activation Oracle (layer 20)

LoRA activation-oracle (verbalizer) on google/gemma-4-26b-a4b-it: reads residual-stream activations extracted at layer 20 (injected at layer 1, additive norm-matched, " ?" placeholder slots) and answers questions about them in natural language.

v3 = the 2026-07-04 retrain on the fixed v2 datasets (supersedes ...-ao-l20-v2, which was trained on data with a corrupted broad tier — see the audit notes in the training repo). One epoch over 302k rows: `cds-jb/synthweb-qa-gemma4-26b-a4b` (99.5k)

• —`cds-jb/cot-qa-gemma4-26b-a4b` (202.7k), both regenerated with: broad tier as a full design pass (own grounded answers, cognitive-state coverage menu), verbatim split anchors, generator-seen-window rows, 30% math cap (cot), majority-preferred suffix (synthweb).

Training

• —wandb japhba-personal/cot-oracle run cjmukq8k (group g4d); final train loss 1.674 @ step 2356.
• —4x B200, HF path (no Unsloth), bf16, LoRA r=64 alpha=128 on all text-stack linears (vision tower excluded) + MoE expert `target_parameters` on every 3rd layer (peft>=0.17 required to load). Effective batch 128 (global 32 x accum 4), lr 2e-4 trapezoid, pure dense/sparse alternating micro-batches (dense = all read positions capped at last 1024; sparse = stochastic E[k]~10), sliced-lm_head loss.
• —In-training verbalization trueness 0.019 (step 30) -> 0.285 (final tick); broad-tier backtracking T 0.50-0.66 (the superseded v2 checkpoint scored 0.02-0.05 broad).
• —ao_config.json in this repo is the full training config (evals derive the train-matched read/injection setup from it).

Usage

Load with peft >= 0.17 (expert target_parameters): PeftModel.from_pretrained(base, "cds-jb/gemma4-26b-a4b-ao-l20-v3"). Extraction/injection utilities live in the training repo (nl_probes/).