Modelpublic

sign/ModernBERT-Large-Instruct-WSD

sourceHugging Faceapache-2.0updated 22d agoView on Hugging Face

0likes881downloads

Model Card

ModernBERT-Large-Instruct-WSD

Word sense disambiguation as multiple choice over WordNet (omw-en:1.4) definitions with a masked-LM answer slot. Used by https://github.com/sign/word-sense-disambiguation (the WSDModernBertForMaskedLM class there adds a 128-way answer-letter decoder; answer_letters.json lists the letters).

This revision (2026-09-04, "c3")

Fine-tuned from answerdotai/ModernBERT-Large-Instruct on: generated sentences for 97k synsets, WordNet's own example sentences (minus a 5,000-example held-out slice), SemCor (222k instances, detokenized) and the Princeton WordNet Gloss Corpus (manual tags). Cosine schedule, label smoothing 0.1, weight decay 0.01, lr 2e-5, 2 epochs, batch 64, fp32 weights with bf16 autocast. Weights are stored in bf16.

Trained with the compact prompt template (the sentence with the target word marked, one definition per option letter, a "none of the above" letter, then [unused0] [MASK]; 16% fewer tokens and ~20% faster than the earlier wording). The repository code from PR #49 on builds exactly this prompt; earlier code/models are not compatible.

Benchmark	previous revision (w4, earlier template)	this revision (c3)
WordNet held-out slice (5,000 examples, seed 42)	78.6%	78.3%
SemEval "ALL" (Senseval-2/3, SemEval-07/13/15; 7,247 instances, any gold key)	80.7%	80.6%
Throughput, one H100, same pipeline	3,614 prompts/s	4,326 prompts/s

The previous revision over-predicted "none of the above" on long natural sentences; this one almost never does. Full recipe and sweep configs: training/ in the repository.