Modelpublic

JonathanMiddleton/daisy-milli-base-v18d.e-tokens258606104576

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face

0likes23downloads

Model Card

DaisyCore — daisy_milli

Model Description

DaisyCore transformer with 26 layers, 14 attention heads, and a model dimension of 1,792. Uses block-causal sliding window attention (window size 2,048) with standard attention implementation.

Architecture

Property	Value
Architecture	DaisyCore
Layers	26
Attention Heads	14
Model Dimension	1,792
Head Dimension	128
Sliding Window Size	2,048
Max Sequence Length	131,072
Vocabulary Size	49,152
Attention Implementation	standard
Value Embeddings	True
Tied Embeddings	False
Skip Mix Mode	linear
Tokenizer	`jonathanmiddleton/daisy`
Dtype	bfloat16
Parameters (total)	2,323,120,245
Parameters (non-embedding)	1,001,914,485
Parameters (embedding)	1,321,205,760

Training Progress

Metric	Value
Checkpoint Step	107,959
Tokens Processed	258.61B (258,606,104,576)
Target Tokens	300.00B (300,000,000,000)
Progress	86.2%
Best Validation Loss	2.07058
Evaluations Performed	1352
Saved	2026-03-08 04:59 UTC

Training Configuration

Optimizers

Optimizer	Parameter Group	Learning Rate
AdamW	head_params	0.003216
AdamW	embed_params	0.1865
AdamW	scalar_params	0.02099
Muon	hiddenmatrixparams	0.025

Schedule & Regularization

Parameter	Value
LR Scale	1.0
LR Schedule	nphaselinear
LR Schedule — beginafterfraction	0.0
LR Schedule — cooldown_fraction	0.0
LR Schedule — floor	0.0
LR Schedule — phases	[{'progress': 0.0, 'scale': 1.0}, {'progress': 0.36117676, 'scale': 0.20527}, {'progress': 1.0, 'scale': 0.1}]
LR Schedule — warmup_fraction	0.0
Gradient Accumulation Steps	8
Muon Warmup Steps	300
Seed	1337

Training Data

Type	Sequence Length	Path
fineweb-edu-shuffled	16,384	`data/fineweb-edu-shuffled/train/*.bin`

Checkpoint Provenance

—Resumed from: JonathanMiddleton/daisy-milli-base-v18d.d

All Hyperparameters

Parameter	Value
window_size	2048
vocab_size	49152
eostokenid	49131
num_layers	26
num_heads	14
model_dim	1792
head_dim	128
maxseqlen	131072
model_spec	daisy_milli
model_class	models.daisy.daisy_core.DaisyCore
target_tokens	100000000000
fullwindowtarget_tokens	3000000000
torchcoordinatedescent_tuning	False
torchinductorconfigmaxautotune	False
overfit	False
full_windows	False
wandb_log	True
wandb_project	milli
wandbrunname	milli_v18d.e
wandb_group	pretrain
resume_checkpoint	JonathanMiddleton/daisy-milli-base-v18d.d
resumetargettokens_override	300000000000
usevalueembeddings	True
usetiedembeddings	False
seed	1337
taskvaldebuglogsamples	False
log_interval	16384
muonwarmupsteps	300
lr_scale	1.0
cooldown_fraction	0.0
lr_schedule	{"name": "nphaselinear", "config": {"cooldownfraction": 0.0, "phases": [{"progress": 0.0, "scale": 1.0}, {"progress": 0.36117676, "scale": 0.20527}, {"progress": 1.0, "scale": 0.1}], "floor": 0.0, "warmupfraction": 0.0, "beginafterfraction": 0.0}}
gradaccsteps	8
vallossevery_tokens	262144000
checkpointwarmuptokens	1
checkpointpern_tokens	524288000
save_checkpoint	True
benchmarks_frequency	2
mmlucachebin_path	data/mmlucache/mmlucache.bin
mmlucachebin_rebuild	False
task_training	False
tracklastn_layers	0