Modelpublic

sign/Ettin-150m-WSD

sourceHugging Faceapache-2.0updated 18d agoView on Hugging Face

1likes560downloads

Model Card

Ettin-150m-WSD

Word sense disambiguation as multiple choice over WordNet (omw-en:1.4) definitions with a masked-LM answer slot, in a 150M-parameter encoder. Used by https://github.com/sign/word-sense-disambiguation (the WSDModernBertForMaskedLM class there adds a 128-way answer-letter decoder; answer_letters.json lists the letters). Prompt: the sentence with the target word marked, one definition per option letter, a "none of the above" letter, then [unused0] [MASK] (the repository's compact template).

This revision (2026-09-08, "u8")

Fine-tuned from jhu-clsp/ettin-encoder-150m (ModernBERT architecture, 22 layers, hidden 768) with soft-target distillation from sign/Ettin-1B-WSD (loss = 0.3 · label cross-entropy with label smoothing 0.1 + 0.7 · T²·KL(teacher ‖ student), T = 2) on two kinds of data at once:

—labeled: generated sentences for 97k synsets, WordNet's own example sentences (minus a 5,000-example held-out slice), SemCor (222k instances, detokenized) and the Princeton WordNet Gloss Corpus (manual tags);
—unlabeled: 1.06M prompts built by the repository's pipeline from 100k Wikipedia sentences, trained on the teacher's distribution only (training/train.py --unlabeled-prompts, prompts from scripts/dump_prompts.py).

Cosine schedule, weight decay 0.01, lr 5e-5, 3 epochs, batch 64, fp32 weights with bf16 autocast. Weights are stored in bf16.

Benchmark	ModernBERT-Large-Instruct-WSD (c3, 395M)	previous revision (y2)	this revision (u8)
WordNet held-out slice (5,000 examples, seed 42)	78.3%	75.9%	75.9%
SemEval "ALL" (Senseval-2/3, SemEval-07/13/15; 7,247 instances, any gold key)	80.6%	80.1%	80.8%
End-to-end pipeline on the same sentences (spaCy trf, gold-position scoring, repository main of 2026-09-08)	75.2%	74.7%	75.3%
Batch pipeline, one H100, steady state	175–250 sentences/s	380–420 sentences/s	380–420 sentences/s

The unlabeled in-domain prompts are what moved SemEval: without them the 150m is saturated at 80.0 ± 0.4 (4–5 epochs, lower learning rates, stronger distillation, the automatically tagged gloss corpus and SemCor+OMSTI were all tried), and continued training on the labeled data alone gives 79.8. Sweep configs: training/sweeps/ in the repository.