Modelpublic

kerzgrr/Monostich

sourceHugging Faceapache-2.0updated 8mo agoView on Hugging Face

1likes176downloads

Model Card

Monostich 100M

A Compact Instruction-Tuned Language Model

![Model](.) ![License](LICENSE) ![PyTorch](https://pytorch.org) ![GGUF](https://github.com/ggerganov/llama.cpp)

A from-scratch LLaMA-style language model pretrained on 16.6B tokens and instruction-tuned on multi-turn chat data

</div>

Overview

Monostich is a ~100M parameter decoder-only transformer trained entirely from scratch. It uses a LLaMA-compatible architecture with modern components (GQA, RoPE, SwiGLU, RMSNorm) and is designed to be lightweight.

—Pretraining: ~16.6B tokens from FineWeb-Edu + Wikipedia
—SFT: Multi-turn instruction tuning on 5 mixed datasets with Llama-3-style chat templates
—Chat template: Llama-3 style — <|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nHello<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n

Model Architecture

Pipeline: Chat Prompt → BPE-32K Tokenizer → LLaMA Decoder (12L) → Token Prediction

Decoder Block (×12)

Each transformer layer contains:

—Grouped Query Attention with RoPE positional embeddings (12 Q heads, 4 KV heads)
—SwiGLU MLP with gated activation (768 → 2048 → 768)
—RMSNorm pre-attention and pre-MLP
—SDPA backend (Flash Attention when available)

Technical Specifications

<table> <tr><td>Architecture</td><td>LLaMA-style Decoder-Only Transformer</td></tr> <tr><td>Parameters</td><td>100,092,672 (~100M)</td></tr> <tr><td>Hidden Dimension</td><td>768</td></tr> <tr><td>Intermediate (MLP)</td><td>2,048</td></tr> <tr><td>Layers</td><td>12</td></tr> <tr><td>Attention Heads</td><td>12 (Q) / 4 (KV) — GQA 3:1</td></tr> <tr><td>Head Dimension</td><td>64</td></tr> <tr><td>Context Length</td><td>1024</td></tr> <tr><td>RoPE θ</td><td>10,000</td></tr> <tr><td>Vocabulary</td><td>32,000 (BPE)</td></tr> <tr><td>Tied Embeddings</td><td>Yes</td></tr> <tr><td>Precision</td><td>bfloat16</td></tr> <tr><td>Weight Size</td><td>~191 MiB (bf16)</td></tr> </table>

Design Choices

<table> <tr><th>Feature</th><th>Description</th><th>Origin</th></tr> <tr><td>RoPE</td><td>Rotary Positional Embeddings for relative position encoding</td><td>LLaMA</td></tr> <tr><td>GQA</td><td>Grouped Query Attention (3:1) for efficient KV cache</td><td>LLaMA-2</td></tr> <tr><td>SwiGLU</td><td>Gated linear unit with SiLU activation</td><td>PaLM, LLaMA</td></tr> <tr><td>RMSNorm</td><td>Root Mean Square normalization (faster than LayerNorm)</td><td>LLaMA</td></tr> <tr><td>Flash Attention</td><td>Memory-efficient attention via PyTorch SDPA</td><td>Dao et al.</td></tr> <tr><td>Weight Tying</td><td>Embedding and LM head share weights</td><td>Standard</td></tr> </table>

Tokenizer

<table> <tr><td>Type</td><td>Byte-Pair Encoding (BPE)</td></tr> <tr><td>Vocabulary</td><td>32,000 tokens</td></tr> <tr><td>Library</td><td>HuggingFace <code>tokenizers</code></td></tr> </table>

Special Tokens

<table> <tr><th>Token</th><th>ID</th><th>Purpose</th></tr> <tr><td><code><|pad|></code></td><td>0</td><td>Padding</td></tr> <tr><td><code><|unk|></code></td><td>1</td><td>Unknown</td></tr> <tr><td><code><|beginoftext|></code></td><td>2</td><td>Beginning of text</td></tr> <tr><td><code><|endoftext|></code></td><td>3</td><td>End of text (document boundary)</td></tr> <tr><td><code><|startheaderid|></code></td><td>4</td><td>Chat role header open</td></tr> <tr><td><code><|endheaderid|></code></td><td>5</td><td>Chat role header close</td></tr> <tr><td><code><|eot_id|></code></td><td>6</td><td>End of turn (generation stop token)</td></tr> </table>

Training Details

Phase 1: Pretraining

<table> <tr><td>Dataset</td><td><a href="https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu">FineWeb-Edu</a> + <a href="https://huggingface.co/datasets/wikimedia/wikipedia">Wikipedia</a></td></tr> <tr><td>Tokens</td><td>~16.6B (~11.6B FineWeb-Edu + ~5B Wikipedia)</td></tr> <tr><td>Context Length</td><td>1024</td></tr> <tr><td>Objective</td><td>Next-token prediction (all tokens)</td></tr> <tr><td>Peak LR</td><td>3 × 10-4</td></tr> <tr><td>Min LR</td><td>3 × 10-5</td></tr> <tr><td>Warmup</td><td>200 steps</td></tr> <tr><td>Schedule</td><td>Warmup → Plateau (10%) → Cosine Decay</td></tr> </table>

Phase 2: Supervised Fine-Tuning (SFT)

<table> <tr><td>Datasets</td><td>Kyoto-Corpus + LMSYS-Chat-1M + Nomi-150M-Chat + Chat-Compilation</td></tr> <tr><td>Context Length</td><td>1024</td></tr> <tr><td>Objective</td><td>Masked cross-entropy (assistant tokens only)</td></tr> <tr><td>Chat Template</td><td>Llama-3 style with header tokens</td></tr> <tr><td>Peak LR</td><td>5 × 10-5</td></tr> <tr><td>Min LR</td><td>5 × 10-6</td></tr> <tr><td>Warmup</td><td>100 steps</td></tr> <tr><td>Schedule</td><td>Warmup → Cosine Decay</td></tr> </table>

Shared Training Config

<table> <tr><td>Optimizer</td><td>AdamW (fused) — β&sub1;=0.9, β&sub2;=0.95, ε=10-8</td></tr> <tr><td>Weight Decay</td><td>0.0</td></tr> <tr><td>Gradient Clipping</td><td>1.0 (global norm)</td></tr> <tr><td>Precision</td><td>bfloat16 autocast</td></tr> <tr><td>Compilation</td><td>Optional <code>torch.compile</code> (max-autotune)</td></tr> <tr><td>Multi-GPU</td><td>Automatic DDP when ≥2 GPUs detected</td></tr> </table>

SFT Datasets

<table> <tr><th>Dataset</th><th>Source</th><th>Notes</th></tr> <tr><td>Kyoto-Corpus</td><td><a href="https://huggingface.co/datasets/Nikity/Kyoto-Corpus">Nikity/Kyoto-Corpus</a></td><td>Multi-turn instruction pairs</td></tr> <tr><td>LMSYS-Chat-1M</td><td><a href="https://huggingface.co/datasets/lmsys/lmsys-chat-1m">lmsys/lmsys-chat-1m</a></td><td>Real-world conversations (redacted rows skipped)</td></tr> <tr><td>Nomi-150M-Chat</td><td><a href="https://huggingface.co/datasets/guus4324343/Nomi-150M-Chat">guus4324343/Nomi-150M-Chat</a></td><td>Synthetic chat data</td></tr> <tr><td>Chat-Compilation</td><td><a href="https://huggingface.co/datasets/aklein4/chat-compilation">aklein4/chat-compilation</a></td><td>Multi-source compilation (system-prompt conversations excluded)</td></tr> </table>

Quick Start

Installation

bash

pip install torch safetensors tokenizers huggingface_hub

Run

bash

wget https://huggingface.co/kerzgrr/monostich/resolve/main/inference.py
python inference.py

The script downloads the model, tokenizer, and config from Hugging Face automatically (cached after first run).

Usage

Interactive chat (default):

bash

python inference.py

Single prompt:

bash

python inference.py --prompt "What is the capital of France?"

Options:

Flag	Default	Description
`--prompt`	None	Single prompt (omit for interactive REPL)
`--temperature`	0.28	Sampling temperature
`--top-p`	0.95	Nucleus sampling threshold
`--max-new-tokens`	context max	Max tokens to generate
`--device`	cuda	Device (`cuda` or `cpu`)
`--seed`	1234	Random seed

Model Family

<table> <tr><th>Model</th><th>Parameters</th><th>Context</th><th>Status</th></tr> <tr><td>Monostich</td><td>~100M</td><td>1024</td><td>Available</td></tr> <tr><td>Couplet</td><td>~200M</td><td>1024</td><td>Training</td></tr> </table>

Limitations

—Scale: At 100M parameters this model is a research prototype, not a production system

File Contents

kerzgrr/monostich/
  README.md                # This model card
  inference.py             # Standalone inference script
  monostich.safetensors    # Weights (bfloat16, SafeTensors)
  config.json              # Model architecture config
  tokenizer.json           # BPE tokenizer (HuggingFace format)
  tokenizer_config.json    # Tokenizer metadata
  special_token_ids.json   # Token ID mapping
  special_tokens_map.json  # Token string mapping

Citation

bibtex

@misc{monostich2026,
  title={Monostich: A Compact Instruction-Tuned Language Model},
  year={2026},
  url={https://huggingface.co/kerzgrr/monostich}
}

Acknowledgments

Built on:

—LLaMA architecture (Meta AI)
—FineWeb-Edu dataset (HuggingFace)
—Wikipedia dataset (Wikimedia)
—Kyoto-Corpus (Nikity)
—LMSYS-Chat-1M (LMSYS)
—Nomi-150M-Chat (guus4324343)
—Chat-Compilation (aklein4)
—PyTorch SDPA / Flash Attention
—HuggingFace tokenizers and hub

A monostich is a poem of a single line — small, but complete.

</div>

Monostich 100M

A Compact Instruction-Tuned Language Model

Overview

Model Architecture

Decoder Block (&times;12)

Technical Specifications

Design Choices

Tokenizer

Special Tokens

Training Details

Phase 1: Pretraining

Phase 2: Supervised Fine-Tuning (SFT)

Shared Training Config

SFT Datasets

Quick Start

Installation

Run

Usage

Model Family

Limitations

File Contents

Citation

Acknowledgments

Decoder Block (×12)