Modelpublic

Nj-1111/Copernicus-Tokenizer

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes

Model Card

Copernicus Tokenizer

Domain-general BPE tokenizer trained from scratch on 3.96 million documents spanning natural language, code, mathematics, and scientific text.

Parameter	Value
Algorithm	Byte-Pair Encoding (BPE)
Vocabulary size	32,685
Merges	32,493
Byte encoding	GPT-2 byte-level (256-char alphabet)
Min frequency	3

Quick start

python

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Nj-1111/Copernicus-Tokenizer")

ids = tokenizer("Hello, world!")
print(ids)

Use in a training loop

python

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained("Nj-1111/Copernicus-Tokenizer")

inputs = tokenizer(
    ["Hello world", "def foo(): pass"],
    truncation=True,
    max_length=2048,
    padding="max_length",
    return_tensors="pt",
)

Special tokens

Token	Role
`<\	endoftext\	>`	BOS / EOS
`<\	unk\	>`	Unknown
`<\	pad\	>`	Padding
`<think>` / `</think>`	Chain-of-thought delimiters
`<\	user\	> `/` <\	assistant\	> `/` <\	system\	>`	Chat roles
`<\	im_start\	> `/` <\	im_end\	>`	ChatML-style markers
`<\	tool_call\	> `/` <\	tool_result\	>`	Tool use

Training data

Domain	Source
Natural language	Wikipedia (multilingual), Common Crawl
Code	The Stack
Mathematics	MATH dataset, arXiv
Science	PubMed, S2ORC

Training code: github.com/Nj-1111/copernicus-tokenizer