Modelpublic

kerzgrr/Haiku-base

sourceHugging Faceapache-2.0updated 12d agoView on Hugging Face

2likes319downloads

Model Card

Haiku-base

Pretrained base model for the Haiku family (~655M)

![Model](.) ![Stage-orange.svg)](.) ![License](LICENSE) ![Architecture](.) ![Demo](https://huggingface.co/spaces/kerzgrr/haiku-demo)

A larger TinyGDN hybrid: Kimi Delta Attention memory plus gated multi-head latent attention

</div>

What this is

Haiku-base is the pretrained (base) checkpoint for Haiku, the ~655M successor to the Tercet family.

—Scales `kerzgrr/Tercet-base` from ~502M to ~655M parameters
—Hybrid Kimi Delta Attention (KDA) recurrent layers + gated MLA (NoPE) full-attention layers
—Own 65,536 BPE tokenizer (not the Tercet 49k vocab)
—This repo is pretrain-only raw text continuation
—Chat / instruction SFT is not released

This base model is for continuation and research. It will not follow instructions reliably.

Model Architecture

Pipeline: Text Prompt → BPE-65K Tokenizer → Haiku Hybrid Decoder (36L) → Next-token Prediction

Hybrid block schedule (×36)

Every 4th layer is gated MLA; the rest are Kimi Delta Attention:

KDA, KDA, KDA, MLA, … (3:1 recurrent-to-attention)

Component	Details
Kimi Delta Attention	Linear-time recurrent memory (`flash-linear-attention`)
Gated MLA	DeepSeek-style latent KV, content-only QK (NoPE), full-rank output gate
MLP	SiTU-GLU
Residuals	Block attention residual
Norm	Zero-centered RMSNorm
Embeddings	Tied input / output

Technical specifications


Architecture	Haiku hybrid (KDA + gated MLA)
Parameters	655,270,488 deployable
Hidden size	1,024
Intermediate (MLP)	3,840
Layers	36 (27 KDA + 9 gated MLA)
Attention	8 heads, Q LoRA rank 512, KV LoRA rank 256
Linear (KDA)	8 heads × 128 dim
Context (trained)	2,048
Max position embeddings	32,768
Vocabulary	65,536 (BPE)
RoPE θ	1,000,000 (partial factor 0.5; used by KDA)
Precision (Hub weights)	bfloat16 EMA
Weight file	`model.safetensors` (~1.22 GiB)

Training (pretrain)


Dataset	FineWeb-Edu (10.13B packed train tokens)
Tokens seen	4,404,019,200
Sequence length	2,048
Objective	Next-token prediction (+ MTP during training; not used at decode)
Optimizer	Hybrid Muon + AdamW — β₁=0.9, β₂=0.95
Peak LR	2 × 10⁻⁴
Warmup	1% of steps
Grad clip	1.0
EMA	Karras power EMA (γ=1.0, p=0.75, max decay 0.9999) — this Hub file is the EMA weights
Checkpoint	optimizer step 8,400
Val loss (EMA)	3.6904 (ppl 40.06)

Install

1) System requirements

—Python 3.10+
—CUDA GPU strongly recommended
—PyTorch with CUDA matching your driver

2) Create an environment

bash

python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux / macOS
source .venv/bin/activate

3) Install PyTorch

Pick the build for your platform from https://pytorch.org. Example:

bash

pip install torch --index-url https://download.pytorch.org/whl/cu124

CPU-only:

bash

pip install torch

4) Install Python deps

bash

pip install safetensors tokenizers huggingface_hub

Flash Linear Attention is installed automatically by `inference.py` on first run (pinned commit + Windows import patches when needed). Git must be on PATH.

5) Download the inference script

bash

curl -L -o inference.py https://huggingface.co/kerzgrr/Haiku-base/resolve/main/inference.py

# or Hugging Face CLI
hf download kerzgrr/Haiku-base inference.py --local-dir .

The script auto-downloads model.safetensors, config.json, tokenizer.json, and the tiny_gdn/ package from this repo.

Quick start

Single prompt (streams tokens):

bash

python inference.py --prompt "The history of computing begins"

Interactive REPL:

bash

python inference.py

Common options:

Flag	Default	Description
`--prompt`	(none)	One-shot continuation; omit for REPL
`--temperature`	`0.8`	Sampling temperature
`--top-p`	`0.95`	Nucleus sampling
`--top-k`	`50`	Top-k (0 disables)
`--max-new-tokens`	`256`	Generation length
`--repetition-penalty`	`1.08`	Repetition penalty
`--context-length`	`2048`	Tokens kept in the window
`--seed`	`42`	RNG seed
`--device`	`cuda` if available	`cuda` or `cpu`
`--no-stream`	off	Print the full completion at once
`--no-bos`	off	Do not prepend `<\	beginoftext\	>`
`--local-dir`	(none)	Use a local snapshot directory

Files

kerzgrr/Haiku-base/
  README.md
  inference.py
  requirements.txt
  model.safetensors
  config.json
  tokenizer.json
  tokenizer_config.json
  special_tokens_map.json
  special_token_ids.json
  merges.txt
  vocab.json
  chat_template.jinja
  tiny_gdn/
    __init__.py
    config.py
    model.py
    haiku_layers.py
    nn_common.py

Limitations

—Base model: not instruction-tuned; may ramble or fail at Q&A format
—Scale: ~655M parameters — research / edge prototype, not a frontier model
—Dependency: requires flash-linear-attention (KDA); not GGUF / llama.cpp compatible today
—Context: trained at 2,048; longer windows are experimental

Model family

Model	Parameters	Architecture	Stage	Hub
Monostich	~100M	LLaMA-style	SFT	`kerzgrr/Monostich`
Monostich-2-base	~150M	TinyGDN hybrid	Pretrain	`kerzgrr/Monostich-2-base`
Monostich-2	~150M	TinyGDN hybrid	SFT	`kerzgrr/Monostich-2`
Couplet-base	~268M	TinyGDN hybrid	Pretrain	`kerzgrr/Couplet-base`
Couplet	~268M	TinyGDN hybrid	SFT	`kerzgrr/Couplet`
Tercet-base	~502M	TinyGDN hybrid	Pretrain	`kerzgrr/Tercet-base`
Tercet	~502M	TinyGDN hybrid	SFT	`kerzgrr/Tercet`
Haiku-base	~655M	KDA + gated MLA	Pretrain	this repo

Citation

bibtex

@misc{haikubase2026,
  title={Haiku-base: A 655M Hybrid KDA + Gated-MLA Language Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Haiku-base}
}

Acknowledgments

—flash-linear-attention (Kimi Delta Attention)
—FineWeb-Edu
—Tercet family: `kerzgrr/Tercet-base`
—PyTorch SDPA / Hugging Face Hub + tokenizers

A haiku is three lines — larger than a tercet, still compact.

</div>