Phase-Technologies/forge-3b-dpo-data

FORGE-3B DPO Preference Data

Tokenized (prompt, chosen, rejected) preference triples for DPO post-training of FORGE-3B, built per the FORGE paper Section 6.2 / Appendix A.2.

This is data preparation output only — no model was trained to produce this.

Stats

• —Total pairs: 0 (paper target: ~200,000)
• —Domains: 0/4
• —Context length: 4096 tokens (paper Appendix A.2, DPO block)
• —Format: unpacked — one (prompt, chosen, rejected) triple per training example
• —Chat template: <|SYS|>...<|/SYS|> <|USR|>...<|/USR|> <|ASST|>...<|/ASST|> (identical to SFT)
• —Tokenizer: CRAYON (xerv-crayon) ONLY — no fallback tokenizer is used, since DPO requires exact token-id alignment with the frozen SFT reference model (paper Sec 6.2)

Domain Breakdown

Usage

import numpy as np
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="Phase-Technologies/forge-3b-dpo-data",
    filename="ultrafeedback/train_shard_0000.npz",
    repo_type="dataset",
)
data = np.load(path, allow_pickle=True)
chosen_full_ids = data["chosen_full_ids"]      # object array of int32 arrays
rejected_full_ids = data["rejected_full_ids"]   # object array of int32 arrays
chosen_loss_mask = data["chosen_loss_mask"]     # 1 = completion token (compute logprob here)
rejected_loss_mask = data["rejected_loss_mask"]

Per FORGE paper Section 6.2 DPO config

• —β = 0.1
• —lr = 5e-7 (constant)
• —batch = 32 preference pairs/step
• —grad_clip = 0.3
• —reference_model = frozen SFT checkpoint