saraelhark/say-my-name-byt5-small

say-my-name-byt5-small

A `google/byt5-small` checkpoint fine-tuned to predict the IPA pronunciation of a name, given the name and the language it should be pronounced in. Note that lang is the pronunciation language, not the name's etymological origin: "de: Michael" asks for the German pronunciation ˈmɪçaˌeːl, "it: Michael" for ˈmajkol. Part of the say-my-name project: build a names+IPA dataset, fine-tune a G2P model on it, and benchmark it.

Byte-level (ByT5) on purpose: names span scripts (Arabic, accents, unusual letter combinations) that trip subword tokenizers, and bytes have no out-of-vocabulary gaps.

Input format

"<lang>: name", plain ISO 639-1 codes e.g. "fr: Amelie", "ar: أرسلان".

from transformers import AutoTokenizer, T5ForConditionalGeneration

tok = AutoTokenizer.from_pretrained("saraelhark/say-my-name-byt5-small")
model = T5ForConditionalGeneration.from_pretrained("saraelhark/say-my-name-byt5-small")

inputs = tok(["it: Giovanni"], return_tensors="pt", padding=True)
out = model.generate(**inputs, max_length=64)
print(tok.batch_decode(out, skip_special_tokens=True))  # -> ['d͡ʒoˈvanni']

Training

• —Base: google/byt5-small; data: saraelhark/say-my-name (24,841 train pairs, 8 languages, split disjoint by (lang, name))
• —8 epochs, batch size 32, lr 3e-4, seed 13; manual PyTorch loop. Deterministic in practice: two independent Colab runs produced identical losses, an identical dev curve and an identical test table.
• —The released checkpoint is epoch 7, not epoch 8. Dev PER bottomed at 0.068 on epoch 7 and rose to 0.077 on epoch 8, so training past that point was actively making the model worse. The exact command and the full per-epoch curve ship next to the weights as train_args.json / train_history.json.

Evaluation

Full test split (2,507 names, no subsetting), scored against every attested variant — a prediction counts if it matches any. Baselines: espeak-ng (rule-based) and CharsiuG2P (same architecture, run zero-shot, which isolates whether fine-tuning on names helped rather than "using a neural G2P at all").

• —micro weights every name equally, so it is dominated by Polish (56% of the data); macro is the mean of the per-language rows. Quote macro when the claim is about languages, micro when it is about names.
• —exact%-ns ignores stress marks and tie bars. The gap between it and the strict column is how much of a score is transcription convention rather than pronunciation.
• —feat-dist is panphon's feature-weighted distance, normalized by name length. panphon ignores stress marks entirely, hence the separate stress% column: the share of names — among those whose references mark stress at all — where stress lands on the right syllable. French and Vietnamese references mark none, so they don't contribute to it.
• —PER, feat-dist and stress% score over canonical segments, so an affricate written dʒ and one written d͡ʒ compare equal. Unreconciled, that single typographic choice costs PER 0.25 on a name like Giovanni, and 24% of dataset rows use the tie bar while the baselines mostly don't. exact% stays strict.

Per-language exact-match (test):

Reproduce from the repo:

uv run python -m pronounce.eval --baseline finetuned \
  --model-dir saraelhark/say-my-name-byt5-small --split test

Correction to an earlier version of this card

An earlier version reported espeak-ng at 7.7% and CharsiuG2P at 16.0% / 0.722. Both were artifacts of how those baselines were invoked, not properties of the systems:

• —espeak-ng: phonemizer's espeak backend deletes stress marks unless with_stress=True. Every Italian, Spanish, Polish and Turkish reference here carries one, so those four languages were pinned at exactly 0.0% exact-match. Correctly invoked, espeak-ng scores 35.0% micro / 30.1% macro.
• —CharsiuG2P: it was run without its mandatory "<iso639-3>: name" language tag, which makes the model emit repeat-until-maxlength garbage (PER ≈ 3.5), and on a "100-name subset" that was really an alphabetical prefix — 57 Arabic + 43 German names and nothing else. It was also tokenized with `addspecial_tokens=True`, where its published snippet says False. Invoked as documented, on the full split, it reaches PER 0.211 — and 89.5% notation-agnostic exact-match on Italian, so it knows the phonemes and simply never writes stress the way this dataset does.

This model's own numbers moved too, for two reasons. The same fairness rules now apply to every system — where the references don't assert stress at all a prediction isn't marked wrong for supplying it, and affricate notation is reconciled before scoring. And the checkpoint itself was retrained on the cleaned dataset and now stops at the best epoch.

Against the previously published checkpoint: micro 87.4% → 88.1%, macro 69.3% → 68.3%. German gained the most (47.6% → 54.1%, eleven more names out of 170), which is what you would expect from deleting 66 syllable-fragment rows that were 3.8% of its training data. Arabic and Vietnamese fell, but by five names and four names on test sets of 57 and 37 — small-sample noise, and the whole of the macro decline. The reason to prefer this checkpoint is not the score, it is that it stops at the measured best epoch and was trained on exactly the dataset that is released here.

Limitations

Trained on a modest, imbalanced dataset (~56% Polish). Low-resource languages score far below the micro-average (Arabic 36.8%, Turkish 45.2%, German 54.1%), tracking their small training share — which is exactly the distance between the 88.1% micro and 68.3% macro figures. Those three languages also have the smallest test sets (57, 31 and 170 names), so single-digit swings there move the macro figure by a point without meaning much; treat macro as ±1.

Polish at 98.9% deserves particular suspicion: Polish Wiktionary IPA is largely template-generated from spelling (1.01 variants per name, against 2.47 for Spanish), so that row may be measuring how learnable a transliteration rule is rather than anything about names.

The strict/ns gap is the other thing to keep in view: on the notation-agnostic column this model scores 89.9% against espeak-ng's 42.9%, so a real part of what fine-tuning bought is convention alignment with this dataset rather than phoneme knowledge. A portfolio/benchmarking model, not intended for production TTS.

Known limitation: the text-level win does not transfer to audio

Feeding this model's IPA to an espeak-trained TTS voice does not produce better speech than feeding espeak's own phonemes — and in the cleanest measurement it is slightly worse.

Method: round-trip the IPA through Piper and a phoneme recognizer (`facebook/wav2vec2-lv-60-espeak-cv-ft`), then subtract the error floor of that same chain measured on espeak's own phonemes. 100 names per language, identical seeded sample for both runs:

A positive signal means this model's IPA fared worse. French is the more trustworthy measurement — median floor 0.200, and 15 of 100 names round-trip with zero error, so the instrument is working rather than saturated.

The cause is notation coupling, not phonetics. Piper's voices are espeak-trained, so espeak phonemes are in-distribution for them by construction, while this dataset's Wiktionary-convention IPA is not. Piper silently discards symbols missing from a voice's table — tie bars, every Vietnamese tone letter, the Spanish lowering diacritic. Six of the eight languages have a recognizer error floor too high to resolve anything at all.

If you are dropping this model in as a G2P frontend, measure the audio rather than assuming the 88.1%-vs-35.0% text result carries. Code and method: github.com/saraelhark/say-my-name.

License

Released under CC BY-SA 4.0, matching its training data rather than the project's code.

The dataset is derived from Wiktionary, which is CC BY-SA 4.0, and ShareAlike is the awkward clause for model weights: whether weights are a derivative work of their training data is unsettled, and an earlier version of this card claimed MIT without addressing it at all. CC BY-SA is the conservative reading, and costs nothing here. The code in the repo stays MIT.

The base model, `google/byt5-small`, is Apache-2.0. The espeak-ng and CharsiuG2P systems in the table above were used only for benchmarking — nothing from either is redistributed here, and no weights were initialized from CharsiuG2P (which declares no license).

Links

• —Dataset: saraelhark/say-my-name
• —Code: github.com/saraelhark/say-my-name