Modelpublic

nthngdy/fabric-1p7B-sft-tulu3-lr1e4

sourceHugging Faceapache-2.0updated 6d agoView on Hugging Face

0likes1.6kdownloads

Model Card

fabric-1p7B SFT (Tulu 3) — lr 1e-4

1.7B models supervised-finetuned on allenai/tulu-3-sft-mixture at lr 1e-4, 2 epochs, seq 16384, step 12138. Pretrained bases are the step-95000 arms from `nthngdy/fabric-1p7B-95k` and `nthngdy/fabric-1p7B-95k-copy`.

One branch per arm — pass revision= when loading. The *_copysink_maskcopy branches carry a custom copy-attention head and need trust_remote_code=True.

branch (revision)	attention	sliding window	val ppl	init
`dense`	vanilla	-	2.349	`188624@95000`
`sw256`	vanilla	256	2.567	`188628@95000`
`hybrid3to1_sw256`	vanilla	256	2.353	`188627@95000`
`dense_copysink_maskcopy`	copy-sink	-	2.314	`201111@95000`
`sw256_copysink_maskcopy`	copy-sink	256	2.378	`199424@95000`
`hybrid3to1_sw256_copysink_maskcopy`	copy-sink	256	2.319	`201112@95000`

python

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained(
    "nthngdy/fabric-1p7B-sft-tulu3-lr1e4", revision="sw256_copysink_maskcopy", trust_remote_code=True)
t = AutoTokenizer.from_pretrained("nthngdy/fabric-1p7B-sft-tulu3-lr1e4", revision="sw256_copysink_maskcopy")

Chat format is ChatML (chat_template.jinja ships with each branch). Weights are bf16. The sibling repo `nthngdy/fabric-1p7B-sft-tulu3` holds the lower-lr (2e-5) SFT arms.