olaverse/diacnet-1.1-gloss

DiacNet 1.1 Ambiguity Lexicon & Gloss Tables

Per-language tables of diacritic-ambiguous words — words whose diacritic-stripped form maps to two or more distinct diacritized variants — extracted from the diacnet-1.1-corpus and annotated with variant type and English glosses.

Built to power the gloss-conditioning feature of diacnet-1.1: at training time, ambiguous words in the input can be annotated with their intended English meaning (e.g. [g: ogun=war]), teaching the model to use user-supplied sense hints when restoring diacritics. Only variants typed sense participate in gloss injection; the other types feed marking-consistency augmentation instead.

Files & loading

One CSV per language (gloss_curate_{lang}.csv), each exposed as a config:

from datasets import load_dataset
yor = load_dataset("olaverse/diacnet-1.1-gloss", "yor", split="train")

or as plain CSVs:

from huggingface_hub import hf_hub_download
import pandas as pd
df = pd.read_csv(hf_hub_download("olaverse/diacnet-1.1-gloss",
                                 "gloss_curate_yor.csv", repo_type="dataset"))

Schema

type semantics

• —`sense` — genuine lexical/grammatical ambiguity: the variants are different words or word-forms whose distinction is recoverable only from context or an external hint (e.g. Yorùbá ogún "twenty/inheritance" vs ògùn "medicine/charm"; Spanish término "term" vs terminó "he/she finished"; Vietnamese sách "book" vs sạch "clean"). These rows drive gloss injection.
• —`spelling_variant` — same lexeme, different marking completeness or legal orthographic alternatives (e.g. Yorùbá ninu vs nínú, both "inside"; Portuguese econômica/económica, Brazilian vs European norm; Italian perché/perchè). Used for marking-consistency training, not glossing.
• —`proper_noun` — names/places with transliteration variance (Nàìjíríà, Söke, Elysée, Antônio/António).
• —`skip` — excluded rows: encoding noise (e.g. Vietnamese forms using ð (eth) instead of đ) or unresolvable low-frequency items.

How it was built

1. 1.Tokenized the full diacnet-1.1-corpus train split per language; grouped words by diacritic-stripped form.
2. 2.Kept groups with ≥2 variants each clearing an absolute frequency floor (≥20) and a minimum share of the group (≥2%) — filtering out OCR/typo tails masquerading as variants.
3. 3.Kept only groups with ≥2 distinctly marked variants (a bare form plus one marked form is marking-completeness noise, not ambiguity).
4. 4.Content words prioritized (stripped length > 3); note the limitation on monosyllables below.
5. 5.type and gloss_en drafted by an LLM (Claude) using per-language strategies: rule-based classification for regular grammatical patterns (Polish case alternation, French participle/present, Spanish stress-accent grammar) plus hand-curated entries for lexical polysemy; Vietnamese glossed fully by hand (399/401 groups); Yorùbá curated conservatively against dictionary-attested tonal minimal pairs.

Per-language statistics

The cross-language pattern is itself informative: Vietnamese ambiguity is near-total (tonal monosyllabic language — stripping tones collapses the lexicon), Spanish/Polish/French ambiguity is mostly grammatical (stress-accent, case, tense distinctions), Italian has essentially none (accent pairs are standard-vs-misspelling), and Yorùbá/Igbo/Hausa corpus variance is dominated by marking completeness rather than polysemy — with a small, high-value core of true tonal minimal pairs in Yorùbá.

Known limitations

• —Annotations are LLM-drafted, not native-speaker-verified. Rows whose gloss contains [REVIEW] (16 total, concentrated in yor/ibo/hau) are explicitly uncertain. A native-speaker pass over Yorùbá and Hausa sense rows is recommended before treating those glosses as ground truth. Report corrections via the community tab.
• —Yorùbá monosyllables are under-represented. The extraction deprioritized stripped forms ≤3 characters as probable function words — but much of Yorùbá's real tonal polysemy lives in monosyllabic verbs (sọ/só, wá/wà, gbà/gbá, bá/bà, lọ/lo, rí/rì, fẹ́/fẹ̀). A supplementary monosyllable table is planned.
• —Frequencies reflect the diacnet-1.1-corpus (FineWeb-2 + Wikipedia after density filtering), not general language use; register skews toward web/encyclopedic text.
• —Vietnamese glosses give the dominant meaning(s) with a disambiguating example compound; many syllables have additional senses not listed.

Provenance & license

Derived from diacnet-1.1-corpus (FineWeb-2, ODC-By 1.0; Wikipedia, CC BY-SA 4.0). Released under ODC-By 1.0. Attribute upstream sources per their terms.

Citation

@dataset{diacnet_1_1_gloss,
  author = {olaverse},
  title = {DiacNet 1.1 Ambiguity Lexicon \& Gloss Tables},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/olaverse/diacnet-1.1-gloss}
}