maximorulli/nanochat-muon-limited

Nanochat d26 Muon Limited

This repository hosts maximorulli/nanochat-muon-limited, a d26 Nanochat decoder-only causal language model with custom Transformers remote code for loading the architecture and tokenizer.

Loading

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("maximorulli/nanochat-muon-limited", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("maximorulli/nanochat-muon-limited", trust_remote_code=True, torch_dtype="auto")

Architecture

• —Decoder-only causal Transformer.
• —26 layers, hidden size 1664, 13 attention heads, 13 key/value heads.
• —Context length 2048 and vocabulary size 32768.
• —RoPE positional encoding, RMSNorm/pre-norm blocks, QK normalization.
• —ReLU-squared MLP.
• —Untied token embedding and LM head.
• —Alternating value embeddings / ResFormer-style value paths; 13 value-embedding layers.
• —Residual and x0 lambda scalar parameters.
• —Linear layers without bias.
• —Logit softcap: 15 * tanh(logits / 15).
• —Total parameters: 1,681,790,292.
• —Estimated training FLOPs/token: 6.573818e9.

Tokenizer

The tokenizer is a custom RustBPE/tiktoken-style tokenizer with GPT-4-like splitting and 32768 vocabulary entries. It was trained with a 2,000,000,000 character cap and a 10,000 document cap.

Special tokens:

Token byte statistics: min 1, max 19, mean 6.6029, std 2.8250.

Base Checkpoint Metrics

• —Saved checkpoint metadata is the source of truth for the BPB value.

Datasets

Base/tokenizer/pretraining:

• —karpathy/fineweb-edu-100b-shuffle, parquet shards shard_00000.parquet through shard_01822.parquet.
• —Train split: all shards except the final validation shard.
• —Validation split: final shard.
• —Packing: BOS-aligned best-fit packing to 2048 tokens.
• —Effective token utilization: 100% packed utilization; approximately 35% of overlong rows cropped at T=2048.

SFT provenance documented for this training series:

• —HuggingFaceTB/smol-smoltalk, train and test.
• —Custom identity conversations, used twice during SFT.
• —cais/mmlu, auxiliary_train for SFT training and all/test for validation/evaluation.
• —openai/gsm8k, main/train for SFT training and main/test for validation/evaluation.
• —Synthetic SimpleSpelling(size=200000, split=train).
• —Synthetic SpellingBee(size=80000, split=train).
• —Synthetic SpellingBee(size=256, split=test).

Evaluation datasets:

• —allenai/ai2_arc: ARC-Easy test and ARC-Challenge test.
• —cais/mmlu: all/test.
• —openai/gsm8k: main/test.
• —openai/openai_humaneval: test.
• —Synthetic SpellingBee test.
• —Tokenizer evaluation categories: news, Korean, code, math, science, FineWeb-Edu train, FineWeb-Edu validation.

Tokenizer Evaluation

Compared with GPT-2:

Compared with GPT-4:

Related SFT And Downstream Metrics

These related downstream/SFT results are included for context and should not be read as direct base-checkpoint metrics unless explicitly stated.

Muon-full-derived SFT, source d26, SFT step 501:

Eval settings: dtype=bfloat16, temperature=0, max_new_tokens=512, top_k=50, batch_size=8.

AdamW-source SFT, base model step 7226, SFT step 501:

• —SFT validation BPB: 0.38646146285770866.
• —ChatCORE/downstream task metrics: not reported.

Muon step-2000-source SFT, base model step 2000, SFT step 80:

• —SFT validation BPB: 0.3852505442606574.
• —Stop reason: reached target validation BPB.
• —ChatCORE/downstream task metrics: not reported.

Metric Availability Notes

• —CORE estimates are not reported for these base checkpoints.
• —Metrics are reported only when they are part of this release.
• —A later AdamW continuation checkpoint is outside the scope of this release.

Intended Use

This is a small research/training artifact intended for experimentation with Nanochat-style training, optimizer comparisons, tokenizer behavior, and lightweight causal language modeling. It has not been aligned for broad production use.

Limitations

The model may produce incorrect, unsafe, biased, or low-quality text. The base checkpoints are not instruction-tuned chat assistants. Context length is 2048 tokens. Generation through the included Transformers implementation is portable but does not include the original training code's custom Flash Attention path.