HauserGroup/ApeTokenizer-SMILES

1---2license: mit3library_name: transformers4tags:5- chemistry6- molecules7- smiles8- ape-tokenizer9- tokenizer10---11 12# ApeTokenizer-SMILES13 14ApeTokenizer-SMILES is an **Atom Pair Encoding (APE)** tokenizer for SMILES15strings, trained on ~2M unique canonical SMILES from ChEMBL 36. It is16the SMILES counterpart to17[ApeTokenizer-SELFIES](https://huggingface.co/HauserGroup/ApeTokenizer-SELFIES),18released alongside19[ModernMolBERT](https://github.com/HauserGroup/ModernMolBERT).20 21APE is a byte-pair-style merging scheme applied to SMILES symbol pieces (atoms,22bonds, ring and branch symbols). Merges are frequency-driven, so a single token23may span structural boundaries (for example `(=O)N(`), matching the APE24behaviour described in Leon et al. — token boundaries are not guaranteed to be25balanced sub-structures.26 27## Tokenizer Details28 29- **Developed by:** Hauser Group, Department of Drug Design and Pharmacology, University of Copenhagen30- **Input representation:** SMILES (canonical)31- **Algorithm:** Atom Pair Encoding (APE) — pair merging over SMILES symbol pieces32- **Vocabulary size:** 138633- **Max merge pieces:** 634- **Min merge frequency:** 300035- **Training corpus size:** 2M unique canonical SMILES (ChEMBL 36)36- **License:** MIT37- **Repository:** https://github.com/HauserGroup/ModernMolBERT38 39| special token | id |40|---------------|----|41| `<s>` (BOS) | 0 |42| `<pad>` | 1 |43| `</s>` (EOS) | 2 |44| `<unk>` | 3 |45| `<mask>` | 4 |46 47## How to Get Started48 49```python50from transformers import AutoTokenizer51 52tokenizer = AutoTokenizer.from_pretrained(53    "HauserGroup/ApeTokenizer-SMILES",54    trust_remote_code=True,55    use_fast=False,56)57 58# A canonical SMILES — here aspirin.59smiles = "CC(=O)Oc1ccccc1C(=O)O"60 61tokens = tokenizer.tokenize(smiles)62print(tokens)63# ['CC(=O)', 'Oc1cc', 'ccc1', 'C(=O)O']64 65inputs = tokenizer(smiles, return_tensors="pt")66print(inputs["input_ids"])67```68 69Feed SMILES directly — no SELFIES conversion is needed. This is a standalone70alternative tokenizer and is **not** the tokenizer used by the released71SELFIES-based ModernMolBERT checkpoints.72 73## Citation74 75```bibtex76@article{madsen_modernmolbert,77  title  = {ModernMolBERT: A ModernBERT Encoder Family for SELFIES Molecular Language Modeling},78  author = {Madsen, Jakob S. and Angelucci, Sara and Hauser, Alexander S.},79  year   = {2026}80}81```82 83The APE algorithm follows Leon et al., *Comparing SMILES and SELFIES84tokenization for enhanced chemical language modeling*, Sci. Rep. 14, 25016 (2024).85