yuanxin112/morphbench-en-task6-definition-v2

MorphBench-EN Task 6 — Complex-Word Definition (expanded v2)

Given a morphologically complex English word, generate its dictionary definition: define word=<word> -> → gloss. Derivations come from UniMorph (eng.derivations.tsv), glosses from Wiktionary.

This is the expanded rebuild of the task5a_definition config in `yuanxin112/morphbench-en` (train 6,651 → 14,971), covering 60 derivational affixes / 32 function labels instead of the original 20 / 13.

Splits

Splits are word-family disjoint (union-find over shared stems). train has 14,971 rows and dev 2,077.

The test side is a fully crossed grid of the two exposure axes — how often the whole word appeared in the pretraining corpus, and how well its base is attested there (base_exposure = substring exposure, cut at 500):

Split names are test_<band>_<tier>, e.g. test_unseen_basehigh.

The headline split is `test_unseen_basehigh`: the model has never seen the word but has seen its base, so it must compose base + affix. The memorization control is test_frequent_basehigh, and

memorization gap = Acc(test_frequent_basehigh) − Acc(test_unseen_basehigh)

Because both cells sit in the same basehigh column, the gap isolates whole-word exposure. Cross-band comparisons should stay inside one tier column. This matters: an earlier version of this dataset bucketed the test side by whole-word frequency alone, leaving base exposure uncontrolled (the headline bucket was 100% high-base but the memorization bucket only 71%). Splitting those buckets by base exposure moved Acc@1 by −0.181 for a BPE tokenizer and +0.052 for a morphological one on the same data — i.e. the uncontrolled axis interacted with the variable such a benchmark exists to compare.

Fields

Known limitations

Please read before using this for eval — these are properties of the source, not bugs to be filtered away:

• —Glosses are lexicographic, not compositional. Wiktionary records attested meaning, so lexicalized entries (coideal → "a particular linear subspace in a coalgebra") are correct but not derivable from base + affix. Roughly 20% of sampled items are not cleanly compositional.
• —The gold gloss usually names the base (~59% of test_unseen_basehigh contain the base as a token) — a dictionary convention. Any metric on this data is partly measuring whether the model can recover the base string from the derived word.
• —Dictionary house style dominates the surface form. Targets like "the state or quality of being X" mean n-gram metrics (ROUGE/BLEU/chrF) award ~0.25–0.30 even when the base slot is completely wrong. Prefer SBERT cosine, or measure base-slot accuracy directly.
• —Single sense per word. 27% of the words have ≥2 usable Wiktionary senses, but only the first is kept, so a model producing a different valid sense is scored wrong. Multi-reference scoring is recommended.
• —Function labels are noisy for polysemous prefixes (sub- below vs almost, over- excessive vs across, in- negation vs inside). The label only affects grouping/reporting, not the gloss target.

Source

Built by scripts/en/wiktionary/26_build_task6_definition.py. UniMorph is CC-BY-SA; Wiktionary text is CC-BY-SA 3.0, so this dataset inherits CC-BY-SA.