gatilin/sparklm-31m-tinystories

SparkLM-31M-TinyStories

This is a SparkLM checkpoint exported into a HuggingFace-compatible directory. It is intended to load through the Transformers remote-code path.

Architecture

• —Model type: SparkLMForCausalLM
• —Parameters: 30,814,720
• —Hidden size: 512
• —Layers: 8
• —Attention heads: 8
• —KV heads: 2
• —MLA: True
• —MoE: False
• —MTP: False
• —Q/K Norm: False
• —Tokenizer source: results/mps31mtinystories_final

Checkpoint Metadata

• —Step: 199
• —Loss: 1.921442563533783

Training loss is not a benchmark score. Publish evaluation results separately before comparing this checkpoint with other public models.

Intended Use

This release is suitable for reproducibility checks, loading smoke tests, and lightweight text-generation experiments. It is not a safety-aligned assistant release unless a downstream alignment and evaluation report is attached.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "gatilin/sparklm-31m-tinystories"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
inputs = tokenizer("Hello SparkLM", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The exported remote code wraps SparkLM's native PyTorch implementation in a transformers.PreTrainedModel interface.

Release Contract

A public release is considered loadable when the repository root includes:

• —config.json
• —generation_config.json
• —tokenizer_config.json
• —tokenizer vocabulary artifacts
• —model.safetensors or pytorch_model.bin
• —configuration_sparklm.py
• —modeling_sparklm.py
• —tokenization_sparklm.py
• —release_manifest.json

The config and tokenizer config should expose auto_map entries so AutoConfig, AutoTokenizer, and AutoModelForCausalLM can load with trust_remote_code=True.

Expert Parallel Training

This model supports end-to-end expert-parallel training via a differentiable all-to-all dispatch/combine primitive (`_AllToAllSingle` autograd.Function). The backward pass through the EP all-to-all is a reverse all-to-all with swapped split sizes, enabling gradient flow across EP ranks.

Files

The exporter writes model.safetensors when the safetensors package is available, falling back to pytorch_model.bin in minimal environments.

Release Validation

Run this before uploading or after cloning the repository:

python -m sparklm.cli.release.validate_hf_release gatilin/sparklm-31m-tinystories --load --generate

Launch a local inference UI against the same release:

python -m sparklm.cli.inference_webui --model gatilin/sparklm-31m-tinystories