durganani60/medical-bpe-16k

🏥 Medical BPE 16k Tokenizer

A domain-specific Byte-Pair Encoding (BPE) tokenizer trained on PubMed abstracts, designed to demonstrate that training domain—not vocabulary size—is the primary driver of tokenization efficiency on specialized text.

🎯 Model Overview

• —Vocabulary Size: 16,000
• —Algorithm: Byte-Level BPE
• —Training Data: 45,000 PubMed abstracts (~180MB text stream)
• —Fairness Baseline: Compared directly against general-bpe (16k vocab, trained on Wikitext-103) as well as modern general-purpose tokenizers (cl100k_base, o200k_base).

Key Benchmarks & Fertility Comparison

Fertility measures average tokens produced per word (lower is better):

Key Takeaways

1. 1.Domain Over Size: medical-bpe-16k beats general-bpe by 0.372 on held-out medical text with identical vocabulary sizes and algorithms.
2. 2.Efficiency vs. Scale: Despite having an order of magnitude smaller vocabulary, a 16k domain-trained tokenizer outperforms 100k–200k scale general tokenizers on domain-specific text.
3. 3.Single-Token Rate: In a spot-check of 20 medical terms, medical-bpe-16k retains 2 as single tokens, whereas cl100k and o200k retain 0.

🚀 Quickstart & Usage

Installation

pip install transformers

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained("durganani60/medical-bpe-16k")

# Test on medical text
text = "The patient presented with acute myocardial infarction and hypercholesterolemia."
tokens = tokenizer.tokenize(text)
print("Tokens:", tokens)
print("Token Count:", len(tokens))