value-generalization/olmo3-7b-v3-all

olmo37b — constv3 all-tenets DPO (RQ3 upper bound)

Full-fine-tuned DPO of the value-neutral SFT base value-generalization/neutral-sft-v3-olmo3-7b@e93fa4d880af487009cb828b93665e88a428cc16 on the entire value-generalization/constitution-v3-dpo union: 196,000 preference pairs spanning all 49 constitution-tenets-v3 tenets (4,000 pairs/tenet, one pair per prompt). This is the RQ3 "train on all the alignment data" upper bound — a single model, not a per-value steerability grid.

Training

• —Base: value-generalization/neutral-sft-v3-olmo3-7b@e93fa4d880af487009cb828b93665e88a428cc16
• —Data: value-generalization/constitution-v3-dpo (train, 196k pairs), shuffled seed 42
• —Objective: DPO (sigmoid, beta 0.1), full fine-tune, 1 epoch (12,250 steps, effective batch 16)
• —LR 5e-6 cosine, warmup 0.1, max_length 2048, bf16 mixed precision, FSDP full-shard (8 GPU)
• —Chat format: olmo3_chatml (pinned template + eos in the exported dir's valuegen_chat_format.json)
• —Recipe: configs/experiments/dpo_v3_all_olmo3.yaml; launcher: scripts/0918/train_v3_all.sbatch
• —Provenance: value-generalization repo @ 669a6d8