Modelpublic

Tomionkkas/edith-250m

sourceHugging Facemitupdated 16d agoView on Hugging Face

0likes319downloads

Model Card

EDITH 250M

This model is not useful on its own. It is the generative half of a retrieval-grounded system: a 250M-parameter GPT trained from scratch, whose job is to write the answers a retrieved Marvel record cannot compose by itself. Most of EDITH's replies never reach these weights at all - they are rendered directly from the record. Loaded alone, this is a small base model with patchy Marvel knowledge, by design: 250M parameters cannot memorise a 296 MB corpus, so the system was built not to try.

Run it: https://github.com/Tomionkkas/edith

Architecture

16 layers, 16 heads, d=1024, vocab 50,257, block 1,024, tied embeddings. Tokenizer: marvel_bpe_50257 (SentencePiece BPE, trained on the corpus).

Files

file	what
`model.safetensors`	stage 3, instruction-tuned. The shipped model, fp16 (508 MB).
`config.json`	the architecture above. The loader reads it.
`stage2/model.safetensors`	stage 2, Marvel continued-pretrain, before instruction tuning
`tokenizer/marvel_bpe_50257.model`	the tokenizer

lm_head.weight is tied to wte.weight and is dropped from the file; re-tie it after loading.

Why fp16, not fp32

Measured against fp32 on 20 real questions with greedy decoding: 20/20 byte-identical answers. fp16 ships at 508 MB against fp32's 1016 MB, for no measured difference in output. Full comparison in the repository's MEASUREMENTS.md.

Why stage 2 is here, and why it is not the one that ships

Stage 2 has the better perplexity on both validation sets (32.71 general / 12.24 Marvel, re-measured, against stage 3's 38.21 / 18.58). It is also a continued-pretrain model: it CONTINUES text, it does not answer questions. Stage 3 trades perplexity for instruction-following, which is the thing a user actually wants. Both numbers are in the repo's MEASUREMENTS.md.

Training data

A 296 MB corpus of 202,171 records derived from Marvel Database (marvel.fandom.com) and English Wikipedia, plus FineWeb-Edu as general-English replay. Published as a dataset: Tomionkkas/edith-marvel-corpus. Maths, science and non-Marvel history were deliberately left out; weakness there is intentional.

Limitations

Listed honestly and at length in the repository's README under What it gets wrong, with the measurements behind each.

Disclaimer

EDITH is an independent project and is not affiliated with, endorsed by, or sponsored by Marvel Entertainment, LLC or The Walt Disney Company.