forge-3b
Datasets
All datasets matching “forge-3b”forge-3b-pretrain-data
FORGE-3B Pretraining Data
Tokenized and packed pretraining data for the FORGE-3B language model.
Stats
Total tokens: 51.4070B
Domains: 10/10
Sequence length: 2048 tokens
Format: .npy shards of shape (N, 2048) with dtype uint32
Tokenizer: CRAYON (xerv-crayon, standard profile)
Domain Breakdown
Domain
Weight
Tokens (B)
Status
fineweb_edu
30%
15.0008
✓
thestack
16%
8.0011
✓
wikipedia
8%
4.2791
✓
openwebmath
8%
3.9654
✓
books
7%… See the full description on the dataset page: https://huggingface.co/datasets/Phase-Technologies/forge-3b-pretrain-data.forge-3b-sft-data
FORGE-3B SFT Data
Tokenized, chat-templated, loss-masked SFT data for the FORGE-3B language model.
Stats
Total tokens (incl. pad): 1.4007B
Domains: 6/6
Sequence length: 4096 tokens
Format: .npz shards with input_ids (uint32) and loss_mask (uint8), shape (N, 4096)
Chat template: <|SYS|>...<|/SYS|> <|USR|>...<|/USR|> <|ASST|>...<|/ASST|>
Tokenizer: CRAYON (xerv-crayon, standard profile) or fallback HF tokenizer
Domain Breakdown
Domain
Weight… See the full description on the dataset page: https://huggingface.co/datasets/Phase-Technologies/forge-3b-sft-data.forge-3b-dpo-data
FORGE-3B DPO Preference Data
Tokenized (prompt, chosen, rejected) preference triples for DPO post-training
of FORGE-3B, built per the FORGE paper Section 6.2 / Appendix A.2.
This is data preparation output only — no model was trained to produce this.
Stats
Total pairs: 0 (paper target: ~200,000)
Domains: 0/4
Context length: 4096 tokens (paper Appendix A.2, DPO block)
Format: unpacked — one (prompt, chosen, rejected) triple per training example
Chat template:… See the full description on the dataset page: https://huggingface.co/datasets/Phase-Technologies/forge-3b-dpo-data.