achand45/qwen2.5-7b-instruct-nla-L20-seed-study

Qwen2.5-7B-Instruct NLA (L20) — seed-influence study

Three complete Natural Language Autoencoder (NLA) training runs that probe whether an NLA's learned "explanation language" is canonical (all training randomness converges to the same solution) or arbitrary (each run coevolves a private code).

The design is a one-factor-at-a-time (OFAT) star anchored on a published run (NLA 1). Each run here differs from that anchor in exactly one source of randomness.

All four train on the same RL parquet; the data order differs only through the trainer's shuffle seed. Rollout sampling is unseeded in every run by construction (vLLM samples without a seed=, and CUDA/continuous-batching nondeterminism compounds on-policy) — this is the noise floor, not a controlled factor.

The runs

• —NLA 2 — rollout-noise replicate. Reuses NLA 1's SFT artifacts verbatim (merged AV/AR plus the iter_0003834 AV adapter as the KL reference) and re-runs the RL phase alone at --seed 0, so the training data order is byte-identical to the anchor's. The only difference is unseedable rollout sampling. NLA 2 therefore measures the noise floor against which the other two factors are read.

• —NLA 3 — SFT LoRA init. A full new SFT of both heads (AV and AR) at --init-seed 1 --data-seed 0, a fresh LoRA merge, then RL at --seed 0. This required splitting the trainer's single --seed — which drove both the LoRA/value-head init (torch) and the data shuffle (numpy) — into --init-seed and --data-seed. That split is what makes this arm single-factor: a different random rank-r LoRA subspace, same SFT data order. SFT quality reproduced the anchor's on both heads: AV held-out val ppl 4.068 (target ~4.07), AR held-out FVE 62.0% (target ~62%).

• —NLA 4 — RL data order. Reuses NLA 1/2's SFT and changes only the RL trainer seed to --seed 1. No reshuffled parquet was needed: the trainer derives its own order from the seed. This matters for comparability — the 128 in-run eval prompts are selected from held-out docs in file order with no RNG, so --seed 1 moves the training order while leaving the eval prompt set untouched. A physical reshuffle would have moved both together and confounded the data-order effect with prompt-set sampling noise.

Result

Every factor came in inside the ~1 pt in-run eval noise floor. All four NLAs land within 1.1 pt of each other on terminal held-out FVE (72.7 / 73.3 / 73.8, anchor ~73%). Neither the SFT LoRA init nor the RL data order moves terminal FVE detectably.

This is a null result on the FVE axis, and FVE was never the question. Four NLAs converging to the same score is perfectly compatible with each having coevolved a private code. The canonical-vs-arbitrary question is answered by an AV/AR swap matrix (score AV from run i against AR from run j; diagonal ≫ off-diagonal ⇒ private codes, flat ⇒ shared language) and by explanation agreement — neither of which is in this release. Treat the FVE table as evidence that all three arms trained successfully, not as a finding.

Caveat on the anchor comparison: NLA 1 ran at --batch-prompts 256 on 8 GPUs; NLA 2/3/4 ran at 252 on 6 GPUs (252 divides 6, 256 does not). NLA 2/3/4 are mutually clean; only comparisons to the anchor carry this difference. Their run_config.yamls are otherwise byte-identical to the anchor's apart from paths.

Layout

nla2/ nla3/ nla4/
  rl/
    iter_000025 … iter_000400/   RL adapter + reference/ (KL reference adapter)
    critic_latest/               critic model + value_head.safetensors
    optim_latest.pt              optimizer state
    run_config.yaml              fully resolved training config
    nla_meta.yaml                dataset / injection-token metadata
  rl.log                         complete training log (all metrics, all eval FVE)
  run_*.sh                       exact launch script

nla3/sft/                        NLA 3 only — the one arm with its own SFT
  av_sft/ ar_sft/                LoRA adapters, iters 1000–3834 + run_config.yaml
  merged/{av_hf,ar_hf}           merged standalone HF models

Checkpoints are every 25 steps to 400. Each iter_* holds the policy adapter and, under reference/, the frozen KL-reference adapter that run was regularized against.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "achand45/qwen2.5-7b-instruct-nla-L20-seed-study",
                                  subfolder="nla3/rl/iter_000400")

NLA 3's merged models need no base model or adapter:

av = AutoModelForCausalLM.from_pretrained("achand45/qwen2.5-7b-instruct-nla-L20-seed-study",
                                          subfolder="nla3/sft/merged/av_hf", torch_dtype="bfloat16")

Explanations are produced by injecting the activation vector at the ㈎ token (id 149705) in layer 20; see nla_meta.yaml for the prompt templates and injection details.

Provenance

• —Anchor run (NLA 1): `Yooniel/qwen2.5-7b-instruct-nla-L20`
• —Training code: `Yooniel/EasyNLA`
• —Base model: `Qwen/Qwen2.5-7B-Instruct`, activations from layer 20
• —Corpus: FineFineWeb (100k docs, 10 positions/doc), public data

Trained on 6× A100-80GB. Each RL run is 400 steps, ~9.5 h.