ismailtasdelen/pinanolm-50m

PiNanoLM-50M

A lightweight ~50M-parameter decoder-only Transformer for Raspberry Pi, ARM, and CPU-only edge inference.

PiNanoLM-50M is Version 2 of the PiNanoLM family — a direct evolution of PiNanoLM-20M. It keeps the same clean, from-scratch PyTorch architecture but scales capacity (hidden 256→384, GELU→SwiGLU, context 512→1024) for meaningfully better generation quality while remaining efficient on edge devices.

The whole family now shares a single, config-driven engine (pinanolm_core), so 20M, 50M, and future variants (100M, 250M, Instruct, Code, Math, Security) reuse the same model, training, inference, export, and benchmark code — no duplication.

Highlights

• —49,972,608 parameters (~50M), decoder-only GPT-style Transformer
• —SwiGLU feed-forward (V2) with a GELU fallback for V1 compatibility
• —RoPE (Rotary Position Embeddings) + RMSNorm + weight tying
• —Flash attention via scaled_dot_product_attention (auto), manual fallback
• —1024-token context (2× PiNanoLM-20M)
• —Shared BPE tokenizer (vocab 32,768) — identical to PiNanoLM-20M
• —CPU-only inference, dynamic INT8, TorchScript, ONNX, and GGUF export
• —Fully resumable training: AMP (BF16/FP16), grad accumulation, grad checkpointing, cosine LR, AdamW, grad clipping, TensorBoard + CSV + optional W&B
• —Modular, typed, tested codebase (20 unit tests, PEP8, structured logging)

Architecture

The family scales by config alone (see pinanolm_core/variants.py):

Repository layout

pinanolm-50m/
  pinanolm_core/      # SHARED engine: config, model, generation, data, utils, variants
  pinanolm_50m/       # 50M preset (Pinanolm50mConfig / Pinanolm50mForCausalLM)
  pinanolm_20m/       # 20M preset — V1 backwards-compatibility over the shared engine
  training/           # train.py (AMP, grad accum, grad ckpt, resume, TB+CSV+W&B)
  inference/          # generate.py (temp, top-k, top-p, typical, rep-penalty, streaming)
  export/             # export_all.py (TorchScript, ONNX, INT8, GGUF)
  benchmark/          # benchmark.py (tok/s, latency, RAM, CPU, load time; 20M vs 50M)
  scripts/            # train_tokenizer.py, preprocess.py, upload_hf.py
  tests/              # unit tests
  examples/           # run_pipeline.py
  config.json  generation_config.json
  tokenizer.json  tokenizer_config.json  special_tokens_map.json
  requirements.txt  LICENSE  README.md

Installation

git clone https://huggingface.co/ismailtasdelen/pinanolm-50m
cd pinanolm-50m
pip install -r requirements.txt

Requires Python 3.12+ (developed/verified on 3.11/3.12), PyTorch ≥ 2.1. CPU-only; CUDA optional (AMP auto-enables BF16/FP16).

Quick start (Python)

import json, torch
from pinanolm_50m import Pinanolm50mConfig, Pinanolm50mForCausalLM
from tokenizers import Tokenizer
from safetensors.torch import load_model

cfg = Pinanolm50mConfig.from_dict(json.load(open("config.json")))
model = Pinanolm50mForCausalLM(cfg)
load_model(model, "checkpoints/model.safetensors")
tok = Tokenizer.from_file("tokenizer.json")

ids = torch.tensor([tok.encode("Once upon a time").ids])
out = model.generate(ids, max_new_tokens=64, temperature=0.9,
                     top_k=40, top_p=0.9, typical_p=0.95, repetition_penalty=1.1)
print(tok.decode(out[0].tolist()))

Training

Fully resumable; resumes automatically from checkpoints/checkpoint_latest.pt.

# 1. (tokenizer is reused from PiNanoLM-20M; retrain only if explicitly needed)
# 2. preprocess corpus -> token shards (shared pipeline)
python scripts/preprocess.py --corpus-dir data/corpus \
    --tokenizer tokenizer.json --seq-len 1024 --out data/tokenized
# 3. train
python training/train.py --config config.json --data data/tokenized \
    --out checkpoints --epochs 2 --batch-size 8 --grad-accum 8 \
    --precision bf16 --grad-checkpointing

Features: mixed precision (BF16/FP16 on CUDA), gradient accumulation, gradient checkpointing, cosine LR + warmup, AdamW + weight decay, gradient clipping, validation loop, TensorBoard + CSV logging, optional Weights & Biases (--wandb), automatic checkpoint resume.

Dataset

The default pipeline uses permissively licensed public-domain books (Project Gutenberg) for reproducible offline training. The shared pinanolm_core.data module also supports Hugging Face streaming sources — FineWeb-Edu, TinyStories, Wikipedia, public-domain books — via scripts/preprocess.py --hf <source> (requires HF Hub access in that environment). For production pretraining, swap in FineWeb-Edu.

Inference

python inference/generate.py --prompt "Explain HTTP." --max-new-tokens 128
python inference/generate.py --prompt "Once upon a time" --stream \
    --temperature 0.8 --top-k 40 --top-p 0.9 --typical-p 0.95 --repetition-penalty 1.1

Sampling controls: temperature, top_k, top_p, typical_p (locally-typical sampling), repetition_penalty, max_new_tokens, plus token streaming (--stream).

Raspberry Pi / edge optimization

python export/export_all.py --config config.json \
    --safetensors checkpoints/model.safetensors --out export \
    --formats torchscript onnx int8 gguf --llama-cpp ~/llama.cpp

Produces:

• —TorchScript — export/model_torchscript.pt
• —ONNX (opset 17, dynamic axes) — export/model.onnx
• —Dynamic INT8 (QNNPACK on ARM) — export/model_int8.pt
• —GGUF — a llama-format HF bridge (export/hf_bridge/) + .gguf files when a llama.cpp clone is provided (PiNanoLM's RoPE+RMSNorm+SwiGLU+tied layout maps onto the llama architecture for convert_hf_to_gguf.py).

Target devices: Raspberry Pi 4, Raspberry Pi 5, Orange Pi, Rock Pi, ARM Linux.

INT8 dynamic-quant latency is only meaningful on the target ARM/RPi device (QNNPACK). On x86 the quantized .pt still exports (smaller), but latency numbers should be measured on-device.

Benchmark

# single model
python benchmark/benchmark.py --config config.json \
    --safetensors checkpoints/model.safetensors
# compare 20M vs 50M
python benchmark/benchmark.py --compare --variant pinanolm-20m --variant pinanolm-50m

Measures model size, load time, peak RAM, CPU utilization, latency (ms/token) and throughput (tokens/sec), and prints a Markdown comparison table (also saved to benchmark_result.json). Run on x86, RPi 4, and RPi 5 to compare across devices.

Tests

python -m unittest discover -s tests

20 tests: exact param count, forward shape, loss computation & decrease, weight tying, SwiGLU activation, flash-vs-manual attention agreement, gradient checkpointing, typical sampling, tokenizer roundtrip, scheduler warmup/decay, and variant-registry scaling.

Backwards compatibility

pinanolm_20m re-exposes the original Pinanolm20mConfig / Pinanolm20mForCausalLM API on top of the shared engine (GELU FFN, ctx 512), so existing V1 code and the shared tokenizer keep working unchanged.

Future variants

The codebase is designed so future models require only a preset entry in pinanolm_core/variants.py: PiNanoLM-100M, PiNanoLM-250M, PiNanoLM-Instruct, PiNanoLM-Code, PiNanoLM-Math, PiNanoLM-Security. The shared engine handles architecture, training, inference, export, and benchmarking for all of them.

License

MIT — see LICENSE.

Citation

@misc{pinanolm50m2026,
  title  = {PiNanoLM-50M: A Lightweight Transformer Language Model for Edge Devices},
  author = {Ismail Tasdelen and contributors},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/ismailtasdelen/pinanolm-50m}}
}

PiNanoLM-50M is for education and research. It is a tiny model and does not provide factual guarantees.