Modelpublic

ceselder/loracle-pretrain-v7-sweep-A-oneq-step1560

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes

Model Card

loracle-pretrain-v7-sweep-A-oneq-step1560

Step-1560 (50% of epoch 1) checkpoint from v7 sweep A "oneq" variant (1 randomly-selected QA per organism, half the data of v7_A). Best-so-far in this run when uploaded; AuditBench peaked at 41.1% here.

Training config

—Base: Qwen3-14B (frozen)
—Interpreter LoRA: rank=256, lora_alpha=32, rslora=True (effective scaling alpha/sqrt(rank)=2.0)
—Direction tokens: svdfixedk16mag7rankfirst, 4480 tokens per LoRA
—Prefix mode: rank_tagged
—Data: ceselder/loracle-pretrain-mix subsampled to 1 random QA row per organism (seed=42 deterministic shuffle+drop_duplicates) → 25k train rows
—Eval: 300 holdout orgs (1 row each)
—Effective batch = 8 (batchsize=1 x gradaccum_steps=8)
—LR = 3e-5, linear schedule, warmup = 500 opt-steps
—Epochs = 1, total 3125 opt-steps; this checkpoint at step 1560 = 50% mark
—examples_seen at this checkpoint: 1560 x 8 = 12,480

Eval numbers at step 1560

Judge: Sonnet 4.6 via OpenRouter, canonical IA-paper rubric.

Set	organisms	any-match
heldout_ia	20	40.0%
triggerrecoveryheldout_ia	20	20.0%
auditbench	56	41.1%
oodmodelsv3	27	29.6%
val/mean_all_evals	-	32.7%

Trajectory of the oneq run so far

step (% epoch)	examples_seen	mean	comments
312 (10%)	2,496	16.3%	first eval
624 (20%)	4,992	27.9%
936 (30%)	7,488	26.1%
1248 (40%)	9,984	30.0%
1560 (50%)	12,480	32.7%	AB peak 41.1%
1872 (60%)	14,976	23.5%	dip
2184 (70%)	17,472	28.2%

Wandb

Training run: https://wandb.ai/adamkarvonen/lora-oracles/runs/wixyr3fi

Layout

—interpreter/ PEFT LoRA adapter
—encoder.pt AO encoder state_dict
—ao.pt AO norm-match hook params
—tokenizer/ Qwen3-14B tokenizer
—loracle_config.yaml Training config snapshot