CoolFace
Modelpublic

ceselder/loracle-pretrain-v7-sweep-A-best-step5625

sourceHugging Faceupdated 5mo agoView on Hugging Face
0likes
Model Card

loracle-pretrain-v7-sweep-A-best-step5625

Best checkpoint from v7 sweep A — step-5625 (90% of epoch 1). This is the highest-scoring loracle we've trained.

Training config

  • —Base: Qwen3-14B (frozen)
  • —Interpreter LoRA: rank=256, lora_alpha=32, rslora=True (effective scaling alpha/sqrt(rank)=2.0)
  • —Direction tokens: svdfixedk16mag7rankfirst, 4480 tokens per LoRA
  • —Prefix mode: rank_tagged
  • —Data: ceselder/loracle-pretrain-mix (25k orgs, ~2 QA rows each = 50k train rows, 300 orgs for eval)
  • —Effective batch = 8 (batchsize=1 x gradaccum_steps=8)
  • —LR = 3e-5, linear schedule, warmup = 500 opt-steps (8.9% of training)
  • —Epochs = 1, total 6250 opt-steps; this checkpoint is at step 5625

Eval numbers at step 5625

Judge: Sonnet 4.6 via OpenRouter, canonical IA-paper rubric (verbatim from paper Appendix J.2, "same specific type of behavior").

Setorganismsany-matchrollout-mean
heldout_ia2045.0%19.2%
triggerrecoveryheldout_ia2030.0%11.2%
auditbench5632.1%9.8%
oodmodelsv32733.3%12.7%
val/mean_all_evals-35.1%-

Full trajectory across the run

step (% epoch)meancomments
625 (10%)24.4%first eval
1250 (20%)22.8%dip
1875 (30%)26.6%climb
2500 (40%)28.3%climb
3125 (50%)34.4%first peak
3750 (60%)29.2%dip
4375 (70%)30.0%recovery
5000 (80%)31.7%climb
5625 (90%)35.1%best
6250 (100%, final)31.4%end of epoch

Wandb

Training run: https://wandb.ai/adamkarvonen/lora-oracles/runs/0n1ymlwa

Layout

  • —interpreter/ PEFT LoRA adapter (load with PeftModel.from_pretrained)
  • —encoder.pt AO encoder state_dict
  • —ao.pt AO norm-match hook params
  • —tokenizer/ Qwen3-14B tokenizer
  • —loracle_config.yaml Training config snapshot