CoolFace
Modelpublic

ASTERIZER/LUNA-Tokenizer

sourceHugging Facecc-by-nc-nd-4.0updated 26d agoView on Hugging Face
0likes67downloads
Model Card

LUNA Tokenizer

The shared tokenizer for the LUNA model family — used by both LUNA-100M and LUNA-300M.

It is the EleutherAI / Pythia-160m (GPT-NeoX) tokenizer, extracted into a standalone repository so both models — and any future LUNA variants — reference a single canonical source.

Details

PropertyValue
Tokenizer classGPTNeoXTokenizer
Vocabulary size50,304 (50,254 + EOS padding to multiple of 128)
BOS token`<endoftext>` (id 0)
EOS token`<endoftext>` (id 0)
UNK token`<endoftext>`
`add_prefix_space`false
UpstreamEleutherAI/Pythia-160m (gpt-neox-20b lineage)
LicenseApache-2.0 (upstream EleutherAI tokenizer)

Files

FileDescription
tokenizer.jsonTokenizer vocabulary + merges (Hugging Face tokenizers format)
tokenizer_config.jsonTokenizer configuration (GPTNeoXTokenizer, special tokens)
config.jsonModel-side config (gpt_neox, vocab_size = 50304) for reference

Usage

python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("ASTERIZER/LUNA-Tokenizer")
ids = tok.encode("Hello, LUNA!")
print(tok.decode(ids))

Related