mjbommar/opengloss-v2.3-definitions

OpenGloss v2.3 — Definitions

The flat definition view: one row for every stored rendition of every live sense's definition, the canonical (neutral, plain) gloss included. This is the reading-level and register grading of OpenGloss v2.3 laid out one row at a time, which is the shape most training and analysis code wants. Join back to opengloss-v2.3-senses on sense_id.

Part of the OpenGloss v2.3 release family — 16 datasets built from one store of 160,724 lexemes and 300,787 live senses, all joinable on derived ids. See Related datasets for the rest.

What's new in v2.3 vs v1.3

1. 1.Schema v3. Every lexeme carries a kind discriminator (simplex, compound, phrasal verb, idiom, proper noun, abbreviation, affix, function word); every sense carries a controlled domain leaf from a fixed ~160-leaf taxonomy instead of free text; every example carries the character span of the headword occurrence inside it.
2. 2.Renditions, not one string. A definition is a set: the canonical one plus rewrites at four reading levels and in four registers, each produced in a single call from the canonical text so they say the same thing at different altitudes.
3. 3.A sense graph, not a word graph. Typed relations resolve to sense ids wherever the target's entry exists in the release, so bank --hypernym--> financial institution points at a meaning rather than at a string.
4. 4.Retrieval data is first-class. Synthetic per-sense queries in eight styles, grounded QA pairs, mined word-in-context pairs, MS MARCO-style triples with graph-derived hard negatives, and graded TREC qrels — all derivable from, and consistent with, the same entries.
5. 5.Derivable identifiers everywhere. v1.3 published a positional id for lexemes and senses (3d_model_noun_0) and nothing below that. v2.3 gives every rendition, edge, query, QA pair and provenance record an id computable from the row alone, and never renumbers: a retired sense is tombstoned, so the ids after it keep their meaning.
6. 6.Per-field provenance. Which model wrote a field, how many tokens it took, what it cost — published as its own dataset.

What changed since v2.2

v2.2 (2026-09-07) added tier 5, the WordNet 3.0 gap. v2.3 adds tier 6: named entities. Every tier before it was selected by word frequency or by WordNet membership, and neither signal ranks a name — a name's importance is a fact about the world, not about a corpus — so v2.2 knew Washington and Lincoln but not George Washington, New York City or World War II. Tier 6 is 15,000 candidates ranked by Wikipedia vital-article level, Wikidata sitelink count, WordNet instance membership and US salience, of which 12,078 became entries. Three schema changes come with it:

• —Entity types are written rather than defaulted. Every proper noun in v2.2 carried entity_type = other, because the two migrations and the kind classifier all wrote that placeholder and nothing ever replaced it. 28,915 proper nouns now carry a real type — person, place, organization, work, event, product, species — taken from the candidate list where it knew one and bought as a single batched verdict where it did not. lexicon and senses gain an entity_type column, and lexicon gains wikidata_qid, the join key for reconciling an entry against Wikidata.
• —Aliases. A name has variants — Lincoln for Abraham Lincoln, the Netherlands for Netherlands, FDR, NASA — and v2.2 had nowhere to put them. A variant that has an entry of its own is now an alias_of edge in opengloss-v2.3-relations (1,267 of them, written by a judged alias pass). The schema also reserves a lexicon.aliases column and alias rows in opengloss-v2.3-inflections for variants with no entry of their own, but no pass populates them yet: `aliases` is empty on every v2.3 row. An alias_of edge is never demoted, pruned, capped or re-judged by the hygiene passes, unlike every other relation type.
• —Two new domain leaves. nature.settlements (cities, towns, villages, neighbourhoods) and law_government.polities (countries, states, provinces, empires, historical polities). A quarter of tier 6 is a settlement or a polity and the taxonomy had no leaf for either; adding a geography root would have been a breaking change to a fixed 15-root vocabulary, so both went under roots that already exist.

Schema. No column was removed or retyped. lexicon gains entity_type, wikidata_qid and aliases; senses gains entity_type; relations gains the alias_of type; inflections gains the alias relation; tier gains the value tier6; and the domain taxonomy gains two leaves (taxonomy version 3).

What changed since v2.1

v2.1 (2026-09-07) added tier 4 and the inflections repo. v2.2 adds tier 5: 43,652 WordNet 3.0 candidate lemmas the earlier tiers lacked — common compounds and technical nouns, adjectives, adverbs and verbs, instances/taxa/organisms excluded — 38,526 of them imported outright, the rest matched against v1.3's own files. The other three changes are about honesty rather than coverage:

• —The lemma fold. lexeme-hygiene (D-79) folded 4,377 inflected-form headwords onto the lemma that already carried their meaning ("databases" onto "database", through the store's own recorded morphology) and retired 172 multiword fragments that began or ended on a function word ("is not", "on top of"). Together with D-76's phantom part-of-speech retirements, 4,549 lexemes store-wide now have every sense tombstoned. A lexeme like that is not counted as a lexeme anywhere in this card or in Stats any more — it has no live sense, so it is not a lexeme by this release's own count — but it is not gone: its surface form still resolves through opengloss-v2.2-inflections, and its lexicon row carries retired = true with a retired_reason explaining why.
• —Provenance on inherited fields. Every field a migration or import wrote, not only what a model wrote from scratch, now carries a migrate-stage provenance record naming where it came from, so "where did this text come from" is answerable by grep rather than by trusting the pipeline that happened to run.
• —A `source` column on lexicon and senses: opengloss-v1.3 for content this project generated or migrated from its own legacy releases, wordnet-3.0 for the tier-5 entries imported directly from Princeton WordNet 3.0.

Schema. No column was removed or retyped. lexicon gains source, retired and retired_reason; senses gains source; tier gains the value tier5.

What changed since v2.0

v2.0 (2026-09-05) covered the frequency-ranked single words. v2.3 adds tier 4: the function words the core ranking had excluded on purpose, and every remaining v1.3 entry at Wikipedia frequency ≥ 10 — mostly multiword compounds ("natural selection", "catalog number"), plus names and rarer single words. That doubles the lexeme count and changes the mix: v2.0 was 99.8% single words; a third of v2.3 is multiword.

Schema. No column was added, removed or retyped in any existing dataset. Three things did change:

• —tier gains the value tier4 (it was core, tier2 or tier3).
• —One new dataset, opengloss-v2.3-inflections: a flat surface-form → lemma lookup (plural, past tense, participles, comparative, superlative, derivations) built from the morphology that the lexicon already carried nested.
• —New provenance note prefixes on tombstones and edges, all reversible and all counted in the store audit: phantom_pos: (a v1.3 part-of-speech block whose glosses defined a component word rather than the compound — 11,440 blocks retired), regen: (relations regenerated for senses that had lost every edge to judging), and retyped: contrast (synonym edges the contrast paragraphs showed to be hypernym or hyponym).

Not row-compatible with v2.0. Lexeme, sense, rendition, edge, query and QA ids are stable for every entry v2.0 had. The derived training sets (retrieval-pairs, retrieval-triples, qrels) re-sample negatives over the larger pool, so their rows differ; and the store-wide quality passes run for v2.3 retired ~3,000 senses of the v2.0 entries (phantom part-of-speech blocks and near-duplicate senses), so those senses are now tombstoned rather than live. Treat v2.3 as a new release, not a delta.

Scope: fewer headwords, far more per headword

v2.3 is not a superset of v1.3. It covers 160,724 of v1.3's 205,988 lexemes — every frequency-ranked single word, plus the compounds and names at Wikipedia frequency ≥ 10 — and spends the difference on depth. If you need breadth of vocabulary, use v1.3; if you need graded renditions, resolved relations, spans, or retrieval supervision, use v2.3.

Key statistics

By tier

• —core — top 10K by composite frequency
• —tier2 — ranks to ~42K
• —tier3 — the rest of the frequency-ranked single words
• —tier4 — stopwords, plus compounds and names at Wikipedia frequency ≥ 10
• —tier5 — the WordNet 3.0 lemmas the earlier tiers lacked: common compounds and technical nouns, adjectives, adverbs and verbs (instances, taxa and organisms excluded); 5,126 from v1.3 files, the rest imported from WordNet
• —tier6 — named entities — people, places, organizations, works and events ranked by Wikipedia vital-article level, Wikidata sitelinks, WordNet instance membership and US salience, which no frequency list ranks

Coverage by tier

The release was built in 6 frequency-ranked passes (core, tier2, tier3, tier4, tier5 and tier6) and they did not all receive the same stages. This table is per-field and per-tier so the gaps are visible rather than averaged away.

Reading levels

Registers

Files

Fields

1,919,007 rows, one row per gloss rendition.

One real row:

{
  "sense_id": "0:noun:0",
  "lexeme_id": "0",
  "headword": "0",
  "pos": "noun",
  "sense_index": 0,
  "domain": "everyday_life.quantity_time",
  "tier": "tier5",
  "reading_level": "neutral",
  "register": "plain",
  "text": "a mathematical element that when added to another number yields the same number",
  "readability_grade": null,
  "is_canonical": true,
  "qa_flags": []
}

Loading it

from datasets import load_dataset

ds = load_dataset("mjbommar/opengloss-v2.3-definitions", split="train")
print(ds)
print(ds[0])

The shards are plain parquet, so nothing forces you through datasets — read them straight, locally or over hf://:

import polars as pl

df = pl.read_parquet("hf://datasets/mjbommar/opengloss-v2.3-definitions/data/train-*.parquet")
print(df.head())

import duckdb

duckdb.sql("SELECT count(*) FROM 'hf://datasets/mjbommar/opengloss-v2.3-definitions/data/train-*.parquet'").show()

A parallel corpus of one definition at four reading levels

import duckdb

rows = duckdb.sql('''
    SELECT sense_id, reading_level, text
    FROM 'data/train-*.parquet'
    WHERE register = 'plain'
    ORDER BY sense_id, reading_level
''').df()
print(rows.head(10))

Identifiers, and how they compose

Every id is derived from structure, never randomly minted, so a consumer can recompute one from a row and join across the whole family without a lookup table. Sense positions are stable across regenerations: a retired sense is tombstoned, not removed, so the indices after it never shift.

An edge id keys on the target's slug, not on the target's sense, so resolving a target never changes the id of the edge that found it.

Reading levels and registers

A rendition is keyed on a (reading_level, register) pair. The canonical rendition of every field is (neutral, plain); everything else is a rewrite of it.

marketing sits on the register axis for convenience but is a genre value rather than a point on the formality scale — worth remembering if you train a formality classifier on this column.

Related datasets

Everything below is built from the same store and joins on lexeme_id / sense_id.

Known limitations

• —It is synthetic. Every string here was written by a language model against a schema, not transcribed from a corpus or checked by a lexicographer. It is well-formed and internally consistent; it is not attested usage, and it will contain confident errors. Do not use it as ground truth about what a word means.
• —Judge scores 70.2/100 (core + tier 2) and 66.7/100 (tier 3). A different model family (Claude Opus) scored fixed 40-entry stratified samples at the close of each build. Sample statistics, not per-entry guarantees, and the judge is itself a model.
• —Relation precision is the weakest axis. Relations were judged for validity and the ones that failed were demoted rather than asserted; symmetric reciprocity finished at 94.2% for synonyms and 94.3% for antonyms, and 4,722 senses were left with no relation at all. Treat a single edge as a hypothesis, not a fact; treat the aggregate graph as usable.
• —`core`, `tier2`, `tier3`, `tier4`, `tier5` and `tier6` are deliberately partial. 160,371 lexemes across core, tier2, tier3, tier4, tier5 and tier6 received the text stages (glosses, examples, encyclopedia) but not the queries, QA pairs, contrasts or register renditions. The coverage table above gives the exact per-field share; nothing is hidden behind an average.
• —The encyclopedia is entry-level. One article per headword, about the headword as a whole. On a polysemous entry it is not a description of any one sense, and it is never used as a positive for one (D-71). It is entry-level reference prose, not a specialist article.

Sources and licences

This release is Creative Commons Attribution 4.0 International (CC-BY 4.0). Of 160,724 lexemes in this release, 40,643 (the tier-5 entries whose source column reads wordnet-3.0) are derived from Princeton WordNet 3.0: their glosses, examples, relations and derivationally related forms, plus WordNet's own capitalisation of the headword (D-78).

The WordNet License permits use, copying, modification and distribution without fee, provided its notice is preserved:

The WordNet License notice is quoted in full on the opengloss-v2.3-lexicon and opengloss-v2.3-senses cards; this repo's WordNet-derived rows are governed by the same terms.

Citation

@misc{bommarito2025opengloss,
  title  = {OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph},
  author = {Bommarito, Michael J., II},
  year   = {2025},
  eprint = {2511.18622},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2511.18622}
}

Tier-5 entries additionally derive from Princeton WordNet 3.0 (D-78):

@article{miller1995wordnet,
  title   = {WordNet: A Lexical Database for English},
  author  = {Miller, George A.},
  journal = {Communications of the ACM},
  volume  = {38},
  number  = {11},
  pages   = {39--41},
  year    = {1995}
}

@book{fellbaum1998wordnet,
  title     = {WordNet: An Electronic Lexical Database},
  editor    = {Fellbaum, Christiane},
  publisher = {MIT Press},
  year      = {1998}
}

License

Released under Creative Commons Attribution 4.0 International (CC-BY 4.0). Attribution to the OpenGloss project is required; commercial use is permitted. See Sources and licences above for the Princeton WordNet License that additionally covers this release's tier-5 entries.