yuanxin112/morphbench-verb-cloze

MorphBench — contextual verb-inflection cloze (EN / DE / FR)

A verb in a natural sentence is replaced by the marker [x]. The prompt gives the lemma and a partial feature bundle; the sentence supplies exactly the missing dimension. The model generates the surface form.

prompt : cloze lemma=<L> context=<sentence with [x]> feats=<partial feats> ->
gold   : the inflected surface form

The benchmark exists to test whether a morphology-aware tokenizer helps a small LM inflect words it has seen little of. That question is only answerable if the items are stratified by how much the model actually saw during pre-training, which is what the 2x2 cell field is for.

Configs and splits

from datasets import load_dataset
ds = load_dataset("yuanxin112/morphbench-verb-cloze", "de")          # "en" | "de" | "fr"
ds["test_lemhigh_formlow"]                  # the headline cell

test is the union of the four cell splits; use whichever is convenient.

What is withheld, and what the sentence has to supply

Two of these are not on the person/number axis.

en/ptcp_vs_past withholds VerbForm. The feats string is identical for both answers; only the sentence decides:

The revolt [x] two weeks to be suppressed.        -> took     (no auxiliary)
the custom group fields are [x] .                 -> hidden   (auxiliary)

It is restricted to the UniMorph lemmas whose participle actually differs from their past; for regular verbs the two are syncretic (walked/walked) and there would be nothing to decide.

de/ptcp is participle formation — the ge-circumfix (ge-macht, ge-fahren), its absence on -ieren verbs (studiert) and inseparable prefixes (verstanden), and the infix on separable ones (auf-ge-kommen). Only tokens whose auxiliary is present are taken, so the sentence licenses the participle reading. German does not get the two-way version: its finite half would be a machte vs machten person/number choice that feats cannot state without leaking the answer, so it would compete with agree_past for the same tokens and strip it of every 3sg item. English has no such conflict because walked is person/number-invariant.

English needs the second axis. Outside be (excluded as an auxiliary) no English verb varies its past form by person or number — walked fills all six slots — so withholding person+number there would leave the subject carrying no information at all. German and French have no such problem (their past cells average 4.0 and 5.1–5.8 distinct forms over the six person/number slots in UniMorph), which is why they use one axis throughout.

Per config:

• —en — agree_pres 527, past 462, ptcp_vs_past 62
• —de — agree_pres 1052, agree_past 888, ptcp 800
• —fr — agree_pres 550, agree_imp 196, agree_ps 154, agree_fut 118, agree_cnd 63, agree_sbjv 38

Frequency stratification (cell)

Two axes, each split low/high at 1 occurrence per million words of that language's pre-training corpus (~11 occurrences in these ~10M-word corpora):

• —lemma_other_freq — how often the model saw this verb through its other forms (the UniMorph paradigm total minus the target form). Excluding the target is what keeps the two axes independent; a paradigm total that included it would make form_high imply lemma_high and collapse the grid into a triangle.
• —form_freq — how often it saw this exact surface string.

A plain seen/unseen split was tried first and rejected: on a 10M-word corpus 93–97% of items land in seen+seen and the interesting cell holds 7–46 rows.

lem_high+form_high is capped at 800 rows per language (it is only a ceiling estimate); the other three cells are taken in full, and rare lemmas are deliberately routed to test rather than train by the split.

Fields

Construction guarantees

• —Lemma-disjoint train / dev / test.
• —Sentence-disjoint too. One UD sentence yields one item per finite verb, and those items used to land in different splits, leaving the test answer verbatim in the training copy of the same sentence (35.6% of English test rows before this was fixed). Verified 0 shared sentences.
• —No hand-written cue word lists. An earlier build recovered the tense from an adverb list containing then / once / back / last / soon / later; 8.3% of English test items had only such an adverb as their cue, i.e. the present tense was equally grammatical and the item had no unique answer.
• —Capped per (phenomenon, lemma) so accuracy is not an average over ~20 frequent verbs, and per (phenomenon, person-number) within each cell so a model cannot score well by always emitting the 3sg form.
• —Blank marker `[x]` occurs 0 times in all three pre-training corpora and in no source treebank, and is split into exactly 3 pieces by every tokenizer tested in all three languages — so the marker itself gives no tokenizer a shorter prompt.

Pre-training corpora the frequencies refer to

The counts are only meaningful for a model pre-trained on those corpora. They are not cleaned of CHAT speaker stubs (*TARGET_CHILD: etc.), which make up 4.7% / 9.7% / 7.7% of the EN / DE / FR corpora — the models saw those tokens, so removing them here would misstate exposure.

Licence and attribution

CC BY-NC-SA 4.0. The sentences are derived from Universal Dependencies treebanks, several of which are CC BY-NC-SA; ShareAlike propagates, so the whole dataset carries the most restrictive licence in the mix.

source_treebank is on every row, so a user who needs a permissively licensed subset can filter to it:

permissive = {"ewt", "atis", "pud", "gsd", "hdt", "rhapsodie", "parisstories"}
ds = ds.filter(lambda r: r["source_treebank"] in permissive)

Known limitations

• —French lem_low+form_high has almost no items (the whole of French UD holds ~20). This is a fact about French, not a sampling failure: a frequent form there almost always belongs to a frequent paradigm. Do not report that cell for French.
• —English lem_high+form_low and lem_low+form_low are small (English paradigms have ~4 forms, so a frequent verb's forms are all frequent).
• —Non-3sg present equals the bare lemma in English and German, so a model that blindly copies the lemma scores a non-trivial floor there (EN ~27%, DE ~23%, FR 0%). Compare configs with that in mind.
• —Scoring should be case-insensitive: a handful of golds come from all-caps or headline text.

Citation

Please cite Universal Dependencies and the individual treebanks listed above.