cds-jb/gemma4-26b-a4b-ao-l20-v2

gemma-4-26b-a4b Activation Oracle — Layer 20 (v2)

A LoRA Activation Oracle (AO) / activation verbalizer for google/gemma-4-26b-a4b-it. It reads the model's own layer-20 residual-stream activations and, in natural language, either discriminates the correct read of an activation (recognition / forced-choice) or describes the underlying cognitive state (verbalization) — e.g. what the model is uncertain about, whether it is lying, second-guessing, or committed to a plan.

Checkpoint: training step 16,000 (~46% of one epoch over 3.35M examples). Supersedes v1 (step 3,200 ≈ 9%). Still mid-training — recognition is developing strongly on several axes; free-text verbalization lags (see Evaluation).

What it does

The AO is a LoRA on the same base model. Run the base over a transcript, grab its layer-20 residual activations at a set of read positions (a "context" window), and inject those vectors into the AO's prompt at reserved " ?" special-token slots (one slot per read position). The AO then answers.

Prompt prefix:

Layer: 20
 ? *  <num_positions>
<your question, e.g. "Is the model second-guessing its reasoning here?">

Each " ?" slot's input embedding is replaced by the corresponding injected activation (injection hook at layer 1).

Training

• —Base: google/gemma-4-26b-a4b-it (MoE, 128 experts) — bf16, attn_implementation=sdpa, gradient checkpointing.
• —Adapter: LoRA r=64, alpha=128, dropout=0.0, on attention (q/k/v/o_proj) + MLP (gate/up/down_proj) across all layers, plus the AO read/inject heads.
• —Read layer: 20 (≈2/3 depth).
• —Data: cds-jb/synthweb-qa-gemma4-26b-a4b + cds-jb/cot-qa-gemma4-26b-a4b, with dense/sparse position alternation (every 2nd sample injects all read positions capped at 1024; the rest a stochastic ~10-position subsample).
• —Optimization: lr 2e-4, effective batch 96 (per-rank 4 × grad-accum 3 × 8 GPUs), plain HF multi-GPU with manual gradient all-reduce (MoE routing makes standard DDP raise on undefined expert grads). Train loss ~10.8 → ~1.15 by step 16k.

Full config is in ao_config.json.

Evaluation (AVBench, at step 16,000)

Evaluated on the 25 model-agnostic AVBench tasks (× narrow/broad × dense/sparse tiers).

Recognition (forced-choice, AUC). The AO reads some cognitive states strongly and many not yet:

Mean AUC across the 100 task×tier slices is ~0.46 — i.e. near-perfect on a handful (factual-lie), solid on several, at/below chance on many still-untrained axes.

Verbalization (free-text, judged trueness 0–1). Still ~0 at this checkpoint on most tasks (a few at 0.03–0.12) — the AO's discrimination is ahead of its description mid-training. Per-item generations + scores are available in the project eval DB.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "google/gemma-4-26b-a4b-it"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "cds-jb/gemma4-26b-a4b-ao-l20-v2")
model.eval()

Injecting activations at the " ?" slots (extract layer-20 residuals from a transcript's read window, overwrite the slot embeddings) requires the harness. The AO is only meaningful with activations injected.

Limitations

• —Mid-training (~46%) — recognition is uneven across tasks; verbalization is early.
• —Tied to google/gemma-4-26b-a4b-it layer-20 activations only.
• —Verbalizations are the AO's interpretation, not ground truth.

License

Governed by the Gemma license of the base model.