Modelpublic

phonemetransformers/ipa-childes-tokenizers

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes

Model Card

CHILDES IPA Tokenizers

Tokenizers for each language in IPA-CHILDES used to train cross-lingual phoneme LLMs in our papers:

—IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modeling
—BabyLM's First Words: Word Segmentation as a Phonological Probing Task

Scripts for creating the tokenizers can be found here. Scripts for training models using these tokenizers can be found here.

To load a tokenizer:

python

from transformers import AutoTokenizer
dutch_tokenizer = AutoTokenizer.from_pretrained('phonemetransformers/ipa-childes-tokenizers', subfolder='Dutch')