pico-lm/pico-decoder-small

Pico Decoder Small

pico-decoder-small is a 65M parameter model in the pico-decoder suite — a lightweight, LLaMA-style decoder-only transformer trained from scratch using `pico-train`. It is designed for transparent and reproducible research into the learning dynamics of language models, and is fully compatible with the pico-analyze toolkit for detailed interpretability analysis.

NOTE: The pico-decoder-small-1 branch contains the full commit history for the training run.

🔧 Model Details

📚 Training

• —Dataset: `pretokenized-dolma`, English-only
• —Training steps: 200,000
• —Batch size: 1024
• —Sequence length: 2048
• —Optimizer: AdamW
• —Learning rate schedule: Linear decay with warmup
• —Compute: 16 A100-SXM4-80GB GPUs

📈 Evaluation and Analysis

This model supports fine-grained analysis using `pico-analyze`. This tool enables researchers to understand how learning unfolds over training, even at modest scales.

We also evaluate perplexity of the model on the `pico-paloma-tinsy` dataset.

📄 Citation

If you use pico-small or any other pico-decoder model in your research, please cite:

@software{pico2025,
    author = {Diehl Martinez, Richard},
    title = {Pico: A Lightweight Framework for Studying Language Model Learning Dynamics},
    year = {2025},
    url = {https://github.com/pico-lm}
}