Modelpublic

mtkachenko/dragonfly-135m-base-v0.1

sourceHugging Facemitupdated 4mo agoView on Hugging Face

0likes4downloads

Model Card

dragonfly-135m

dragonfly-135m is a 135M-parameter Llama-based decoder-only language model trained from scratch on 2.7B tokens of synthetic data. Training followed the Chinchilla compute-optimal recipe (20 tokens per parameter), using a single pass over Cosmopedia-v2 with packed sequences and per-document attention masking. It is not a particularly interesting model except for the fact that it exists; it serves as a baseline.

Model Details

Property	Value
Architecture	Llama (`LlamaForCausalLM`) — SmolLM2-135M
Parameters	~135M
Context length	4,096 tokens

Training

Data

Split	Dataset	Subset	Size
Train	HuggingFaceTB/smollm-corpus	cosmopedia-v2	~2.7B tokens
Validation	HuggingFaceFW/fineweb-edu	sample-10BT (1k docs)	held-out

Training sequences are packed using a best-fit algorithm with per-document attention masks to prevent cross-document attention leakage.

Hyperparameters

Hyperparameter	Value
Target tokens	2,700,000,000
Steps	5,150
Tokens per step	524,288
Per-device batch size	32
Gradient accumulation	4
Effective batch size	128 sequences / 524,288 tokens
Peak learning rate	5e-4
LR schedule	Cosine decay, 10% linear warmup (515 steps)
Weight decay	0.1

Final metrics:

—Train loss: 2.259
—Eval loss (FineWeb-Edu, 1k docs): 4.381
—Training time: ~53.4 hours
—Average step time: ~41.7 s/step

Infrastructure

—Hardware: GB10
—Framework: HuggingFace Transformers + PyTorch

Limitations

This is a raw base model with no instruction tuning or safety alignment. It may produce harmful, inaccurate, or incoherent content. Use with caution.

Third-Party Licenses

The model architecture configuration and tokenizer are derived from SmolLM2-135M by HuggingFace, licensed under the Apache 2.0 License. See THIRD_PARTY_LICENSES for details. The model weights are original and released under MIT.