yuanxin112/wiktionary-morph

Wiktionary Morphology Dataset (German / English / French)

A cleaned, morphologically-annotated dictionary of content words built from Wiktionary, intended as raw material for designing morphology-aware tokenizer benchmarks. There is one record per word (parts of speech merged): each record carries the word's POS list, frequency signals, a morphological-structure label, the full inflectional paradigm, definitions with examples, and translations. The schema is flat (all fields at the top level, like the underlying wiktextract data).

from datasets import load_dataset
de = load_dataset("yuanxin112/wiktionary-morph", "de", split="train")
en = load_dataset("yuanxin112/wiktionary-morph", "en", split="train")
fr = load_dataset("yuanxin112/wiktionary-morph", "fr", split="train")

Data fields

One record per word. A word that is several parts of speech (e.g. run, blackboard, manger) is a single record; pos lists every part of speech, and each sense and inflected form is tagged with the pos it belongs to.

morph_type

morph_type reflects Wiktionary's etymological analysis. A word borrowed as a whole (e.g. editor < Latin editor) is labelled simple, even if it looks like edit + -or. When POS readings disagree, the most structured label wins (compound > derivation > simple).

Frequency signals — why several

• —freq answers "how often does this word's paradigm occur in our training data?" — used to mark seen vs. unseen items for generalization splits. freq == 0 is kept on purpose: unseen derived/compound words are the most valuable generalization test cases.
• —citation_freq is the narrow headword-form count.
• —zipf answers "is this a word people actually use, anywhere?" — based on Wikipedia, subtitles, news, books and web text combined, so it is robust to gaps in the training corpus.

How the data was filtered

Starting from the full Wiktionary edition, an entry is dropped if it matches any of:

1. 1.Not the target language (lang_code).
2. 2.Part of speech is not a word category — removes proper names (pos = name), multi-word phrases, abbreviations, symbols and bound affixes (prefix/suffix). Open- and closed-class words are kept.
3. 3.Surface form contains non-letters / whitespace (non-words, multi-word strings).
4. 4.Inflected-form entry (tagged form-of, or a French/German/English inflection gloss such as "Féminin pluriel de…", "plural of…") — inflection is instead embedded inside inflection_forms.
5. 5.Alternative spelling / variant (alt_of, or a "variant/alternative spelling of…" gloss).
6. 6.Loanword / transliteration etymology and low frequency (freq < 5) — naturalized loans (e.g. Computer, Restaurant) are kept.
7. 7.Rare / dialectal / obsolete — only if all senses are tagged rare/dialectal/regional/uncommon/obsolete/archaic/dated/nonstandard.
8. 8.Science / bio-chemical — only if all senses carry a chemistry/biology/medicine/botany/zoology/anatomy/… topic.
9. 9.Never attested in general corpora — wordfreq Zipf ≤ 0 (removes ultra-rare/technical junk such as alectinib, hexacosanol).

Proper names are handled implicitly: Wiktionary stores them as separate pos = name entries, so dropping those still keeps homographic common words (e.g. the eel Aal).

Provenance

• —Built from the kaikki.org wiktextract machine-readable dumps of Wiktionary (German edition 2026-05; English-language extract 2026-06; French edition 2026-06).
• —zipf values from the wordfreq library.
• —freq / citation_freq from a BabyLM training corpus per language — English ≈ 10M words, German and French ≈ 100M words (French = BabyLM-community/babylm-fra).

Limitations

• —decomposition is provided for English only and comes solely from Wiktionary's explicit affix/compound templates (high precision; present for ~55% of English derivations/compounds, null otherwise). German and French omit decomposition; use etymology_text or an external resource (e.g. UniMorph).
• —hyphenation is unavailable for French.
• —Coverage and labels are only as good as Wiktionary; rare/technical entries may be imperfectly tagged.
• —The dataset is a snapshot of the dump dates above, not live Wiktionary.

License

Derived from Wiktionary, licensed under CC BY-SA 4.0 (and GFDL). Released under CC BY-SA 4.0; attribute Wiktionary and its contributors.

Chinese (zh) — 41,059 words

Built 2026-09-23 from the zh.wiktionary kaikki extract (lang_code=zh, 漢語 sections; dump dated 2026-09). Same pipeline as de/fr (native-edition, multi-lemma gloss-novelty merge), with Chinese specifics:

• —Words are ≥2-character BMP Han strings. Single-character (汉字) entries are NOT records — characters are decomposition material only.
• —The dominant 釋義-header format maps to pos="unknown" (~159k records); only the ones with definitional glosses are kept (191k gloss-less stubs dropped). Gloss-less known-pos records are kept, as in de/en (<2%).
• —freq/citation_freq are null (no corpus wordlist yet); zipf is wordfreq zh (covers traditional lemmas natively).
• —inflection_forms holds script variants (首页, alternative forms), not inflection — Chinese has none.
• —pinyin (zh-only column): first Mandarin Pinyin reading (81% of words).
• —decomposition (zh-only column): compound parts from etymology X／x (pīn) + Y／y (pīn) patterns, tiling-validated against the surface word (英語 = 英 + 語). 70 words.

Turkish (tr) — 12,556 words

Built 2026-09-23 from the tr.wiktionary kaikki extract (lang_code=tr; dump dated 2026-09). Same pipeline as de/fr, with Turkish specifics:

• —pos, sense tags and topics are English-normalized by the extractor; inflection_forms carries full declension/conjugation tables (verbs ~371 forms, incl. periphrastic comparatives daha/en X).
• —freq/citation_freq are null (no corpus wordlist yet); zipf is wordfreq tr.
• —morph_type comes from bare X + Y / X + -EK etymology segments and tiling-validated parenthesised groups (gözlük = göz + -lük); Turkish form-of pages (X sözcüğünün/fiilinin/… çekimi) are dropped, with their inflectional content living in the lemma's inflection_forms.