Apppublic

suku9/smiles-tokenizer-package

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes

App README

SMILES Tokenizer

This is a custom tokenizer for SMILES (Simplified Molecular Input Line Entry System) strings.

Installation

bash

pip install git+https://huggingface.co/suku9/smiles-tokenizer-package

Usage

python

# Basic usage
from smiles_tokenizer import SmilesTokenizer

tokenizer = SmilesTokenizer()
smiles = "CC(=O)OC1=CC=CC=C1C(=O)O"  # Aspirin

# Tokenize
tokens = tokenizer.tokenize([smiles])[0]
print(tokens)

# Encode
encoded = tokenizer.encode([smiles])[0]
print(encoded)

# Use with GPT-2
from smiles_tokenizer.utils import prepare_for_gpt2

model, tokenizer_wrapper = prepare_for_gpt2(tokenizer)

# Now you can use it like a regular Hugging Face tokenizer
inputs = tokenizer_wrapper(smiles, return_tensors="pt")
outputs = model(**inputs)

Features

—Specialized for SMILES strings
—Compatible with Hugging Face's transformers library
—Designed to work with GPT-2 models
—Preserves all functionality of the original SMILES tokenizer