Modelpublic

ceselder/loracle-pretrain-v7-sweep-A-oneq-final-step3120

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes

Model Card

loracle-pretrain-v7-sweep-A-oneq-final-step3120

End-of-epoch (final) checkpoint from v7 sweep A "oneq" variant (1 randomly-selected QA per organism, half the data of v7_A).

Training config

—Base: Qwen3-14B (frozen)
—Interpreter LoRA: rank=256, lora_alpha=32, rslora=True (effective scaling alpha/sqrt(rank)=2.0)
—Direction tokens: svdfixedk16mag7rankfirst, 4480 tokens per LoRA
—Prefix mode: rank_tagged
—Data: ceselder/loracle-pretrain-mix subsampled to 1 random QA row per organism (seed=42 deterministic shuffle+drop_duplicates) → 25k train rows
—Eval: 300 holdout orgs (1 row each)
—Effective batch = 8 (batchsize=1 x gradaccum_steps=8)
—LR = 3e-5, linear schedule, warmup = 500 opt-steps
—Epochs = 1, total 3125 opt-steps; this checkpoint at step 3120 = last eval cycle (100%)

Eval numbers at step 3120 (final)

Judge: Sonnet 4.6 via OpenRouter, canonical IA-paper rubric.

Set	organisms	any-match
heldout_ia	20	40.0%
triggerrecoveryheldout_ia	20	15.0%
auditbench	56	30.4%
oodmodelsv3	27	40.7%
val/mean_all_evals	-	31.5%

train/loss: 0.62 (low — well-converged) val/loss: 1.95

Full trajectory of the oneq run

step (% epoch)	examples_seen	mean	comments
312 (10%)	2,496	16.3%	first eval
624 (20%)	4,992	27.9%
936 (30%)	7,488	26.1%
1248 (40%)	9,984	30.0%
1560 (50%)	12,480	32.7%	best (AB peak 41.1%)
1872 (60%)	14,976	23.5%	dip
2184 (70%)	17,472	28.2%
2496 (80%)	19,968	(filled in by run)
2808 (90%)	22,464	(filled in by run)
3120 (100%, final)	24,960	31.5%	end of epoch

Comparison vs v7_A (2 QA/org, 50k rows)

v7_A best (step-5625): mean 35.1% (heldout 45, trigger 30, AB 32.1, ood 33.3) oneq final (step-3120): mean 31.5% (heldout 40, trigger 15, AB 30.4, ood 40.7)

oneq has notably stronger oodmodelsv3 (40.7% vs 33.3%) but weaker heldoutia + triggerrecovery.

Wandb

Training run: https://wandb.ai/adamkarvonen/lora-oracles/runs/wixyr3fi

Layout

—interpreter/ PEFT LoRA adapter
—encoder.pt AO encoder state_dict
—ao.pt AO norm-match hook params
—tokenizer/ Qwen3-14B tokenizer
—loracle_config.yaml Training config snapshot