Unseen1980/daedalus-checkpoints

Daedalus-150M

A 150M-parameter language model built for CPU inference. Two thirds of its layers are short convolutions with a fixed-size state instead of attention, so decoding does not slow down as the context grows.

Trained from scratch on 59.9B tokens. Code and paper: unseen1980/daedalus.

Quick start

brew install llama.cpp   # or build from ggml-org/llama.cpp
hf download Unseen1980/daedalus-checkpoints instruct/model-q4_0.gguf --local-dir ./daedalus

llama-cli -m ./daedalus/instruct/model-q4_0.gguf -cnv \
  --temp 0.8 --top-p 0.9 --repeat-penalty 1.15

Pass sampling flags. llama.cpp defaults --repeat-penalty to 1.0, i.e. off, and this model will loop on a repeated token without it.

Files

The base model deliberately carries no chat template. Giving one to a base model makes llama.cpp wrap prompts in markup it never saw during training, which produces fluent but unrelated output. Use plain prompts, or llama-completion.

Results

Five-task mean over HellaSwag, ARC-Easy, PIQA, OpenBookQA and WinoGrande, with every peer re-scored on the same harness rather than quoted from its paper.

Validation bits-per-byte 0.8685 over 645M held-out tokens.

SmolLM2-135M stays ahead on quality — conceded in advance. The trade this model makes is speed.

Speed

CPU decode, 4-bit, 8 threads, against a parameter-matched all-attention twin trained on identical data:

The trend is the result. At an empty context the hybrid has nothing to gain — its advantage is the key–value cache it does not keep. Against an external 135M peer the same pattern reaches 2.08× at 2048 tokens.

Per token of context this model reads 6,144 bytes of cache against a 24-layer all-attention model's 12,288 — half. At 2048 tokens that is 12.6 MB re-read per generated token instead of 25.2 MB.

Architecture

18 blocks, d_model 768, vocab 49,152, context 2048

block:  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18
type:   C  C  C  C  A  C  C  A  C  A  C  A  C  A  C  C  A  C

A = full attention (6)      GQA, 12 query heads / 4 KV heads
C = short convolution (12)  depthwise, kernel 3, fixed 2-step state

Tied embeddings, 2048 FFN, RoPE θ=1e6. Q4_0 chosen for ARM kernel speed rather than its error curve.

Training

59.9B tokens over a 16.9B-token corpus (~3.5 epochs, capped at 4 per source) of public English data weighted toward educational text: FineWeb-Edu 37.5%, DCLM-baseline 22.5%, Stack-Edu 9%, FinePDFs-Edu 8%, FinePhrase 7%, Cosmopedia-v2 5%, FineMath + InfiWebMath 6%, FineWiki-en 3%, dialogue 2%.

Muon on weight matrices, AdamW on embeddings and norms. WSD schedule with linear decay to zero over the final 45%. One RTX 5090, ~$46 of GPU time.

Post-training: SFT on smol-smoltalk, then one DPO round on UltraFeedback.

Limitations

• —English only, 2048-token context, single seed.
• —4-bit costs ~6% perplexity, not the ~2.5% intended — quantisation-aware training was built and validated, then crashed on activation and never ran. The f16 files let you re-quantise without retraining.
• —~48% of convolution channels are dead (13.6M inert parameters). They cannot be pruned at export: llama.cpp shape-checks those tensors against the model width.
• —Vocabulary is oversized at 49,152 — inherited from a tokenizer chosen for a distillation plan that was cancelled. Scaling laws suggest 24–32k here; it costs 23% of parameters to a lookup table.
• —Mixture skew 10.42 against a 10.0 pre-registered limit, from training 59.9B tokens on a 16.9B corpus.
• —It is a 150M model. It writes fluent, plausible text and gets many facts wrong. The right reference class is GPT-2 124M.

Citation

@misc{koutsiaris2026daedalus,
  title  = {Daedalus-150M: A Convolution--Attention Hybrid Designed for CPU Inference},
  author = {Christos Koutsiaris},
  year   = {2026},
  url    = {https://github.com/unseen1980/daedalus}
}