CoolFace
Modelpublic

rockerritesh/smollm2-135m-fineweb-edu-200m

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face
0likes19downloads
Model Card

smollm2-135m-fineweb-edu-200m

A 135M parameter language model trained from scratch on FineWeb-Edu.

Model Description

PropertyValue
ArchitectureDeep & narrow Llama-style architecture with Grouped Query Attention
Parameters135M
Layers30
Hidden size576
Attention heads9 (3 KV, GQA)
Context length1,024 tokens
Vocab size49,152
Final loss5.4439
Final perplexity231.3

Training Details

This model was trained from scratch (random initialization) for research purposes.

Training Data

  • —Dataset: FineWeb-Edu (sample-10BT split)
  • —Total tokens: ~200M (streamed)

Training Configuration

HyperparameterValue
Batch size4 sequences
Gradient accumulation8 steps
Effective batch32,768 tokens/step
Total steps6,103
Learning rate6e-4 (cosine decay to 6e-5)
Warmup steps200
OptimizerAdamW (beta1=0.9, beta2=0.95)
Weight decay0.1
Gradient clipping1.0
PrecisionMixed (AMP fp16)
HardwareNVIDIA T4 GPU (16GB)

Training Curve

  • —Initial loss: ~10.9 (random)
  • —Final loss: 5.4439 (perplexity 231.3)

Limitations

  • —Not for production use: Trained on only 200M tokens (produces rough text)
  • —No instruction tuning: Base causal LM, not a chat model
  • —English only: Trained on English FineWeb-Edu data

Related Models