CoolFace
Modelpublic

Neon-tech/Tokenizers

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face
0likes
Model Card

NeonTech Tokenizers

Custom multimodal tokenizer family for the NeonTech model series.

Available Versions

  • —v1 — 131,072 vocab, DeepSeek V4 base + Igbo/Japanese/Arabic extensions, 80 special tokens
  • —See the v1 branch for full details on special tokens and architecture.

Usage

python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(“Neon-tech/Tokenizers”, revision=“v1”)