cmeister/unilid-1940

unilid-1940

UNILID language-identification model covering 1,940 language-script combinations, trained on 60M samples from the GlotLID-C corpus. This repository distributes the trained per-language token probability tables as two .unilid files whose stored weights are identical: a version-2 file with the calibration artifact bundled in (per-language decision thresholds, training-line counts, and the calibration constants), and a version-1 file with no calibration bundled.

Code: https://github.com/Ahmetcanyvz/UNILID (the release branch, tag v0.3.0; installation instructions in the repository README).

These weights require UNILID 0.3.0 or later. An earlier version loads them without an error and then applies no unseen-token constant at all, while printing a line that reports the constant as applied. The constant is applied to the lowest-valued entries of each row. In these weights those entries are the four special tokens, which are set to the training floor, rather than the unseen real tokens. Measured on the file published here, UNILID 0.2.1 modifies 0 of the 1,940 rows where 0.3.0 modifies 1,655.

Files

sha256:

135404c834e9e07435b99551c1c3a570cf3b2ac94cff6c26691e90796381dc91  unilid-1940-calibrated.unilid
31c3d956db7b00c939c4985c86a82a8e8d1af963f8cf3921cebaab257d0d74fd  unilid-1940.unilid
1ef3063b9f9a2a04d2997b8c762d035cf52a33dbc613ccf57567c5f81638b174  calibration.json

Which of the two .unilid files to download

The two files are the same generation of weights. The sha256 of the float32 weight matrix is a4aeff199464032c223ae7c77eaa6f128307180f6758e28b1f8e8cd7c985662e in both, and the sha256 of the language list is 00ad6a35b85c3b0b3816598c86534075cdc57f34e50d7e99bc7c74a379e53e19 in both. Only the container differs.

unilid-1940-calibrated.unilid loads in either inference mode on its own, because calibrated=False reads the same base weights, so one download is enough for most uses. unilid-1940.unilid is the file for base inference without the bundled 161 KB of calibration data, and for applying a calibration artifact of your own through the calibration= argument. It bundles no calibration, so loading it with default arguments raises UnilidCalibrationError; pass calibrated=False or a calibration= path.

What changed on 2026-08-24

unilid-1940-calibrated.unilid and calibration.json were replaced in place. The filenames did not change, so a copy downloaded before 2026-08-24 has the same name and different contents. unilid-1940.unilid was added on the same day and has no earlier version in this repository.

The per-language training that produced the 2026-08-11 release assigned each of the four special tokens a probability of 1/5 in every row, so every real token's probability was one fifth of what it should have been. No special token's stored value is read during scoring: the scorer takes its unknown-token score from a single model-wide constant, and <s>, </s> and <pad> are reachable only by text containing those literal substrings. That fifth of the mass was therefore unusable. In the files published now, every real token is higher by log(5) = 1.6094 nats and the four special tokens are set to log(1e-12).

The container format does not record which of the two generations a file belongs to, because its header encodes only version 1 against version 2. To check a copy you already have, sum the probabilities of the real tokens in any row. The sum is 1.0 for the files published now and 0.2 for the 2026-08-11 release. UNILID 0.3.0 and later measure this at every load and print the range over all rows.

The 2026-08-11 files remain downloadable from revision 8d4044d2b69429e16ce256bde6acfa0c02e68203 of this repository, for reproducing results published against them.

Measured effect of the replacement, base mode, on the 250,000-line held-out subset of the GlotLID-C test pool: 1,807 of the 250,000 predictions change, 699 gold-correct predictions gained and 669 lost, and accuracy moves from 0.9603 to 0.9604. Over the full test pool, macro F1 moves from 0.929 to 0.933 for base inference and from 0.957 to 0.956 for calibrated inference. The reason to use these weights rather than the 2026-08-11 ones is that their stored probabilities are the ones the training procedure was meant to produce, not a measured gain.

Usage

from unilid import load_model

model = load_model("unilid-1940-calibrated.unilid")   # calibrated inference (default)
lang, tokens, score = model.predict("The quick brown fox jumps over the lazy dog.")
print(lang)  # 'eng_Latn'

# Base (uncalibrated) inference from the same file:
base_model = load_model("unilid-1940-calibrated.unilid", calibrated=False)

# The version-1 file, which bundles no calibration:
base_model = load_model("unilid-1940.unilid", calibrated=False)
base_model = load_model("unilid-1940.unilid", calibration="calibration.json")

Requires the UNILID package with its Rust tokenizers extension built (installation instructions in the repository README).

Calibration

Calibrated inference applies two corrections, specified in the UNILID paper. The first is a shared constant c = -17 for unseen-token log-probabilities, applied one-sided and with no renormalization afterwards. The second is a re-examination of low-margin predictions into two groups of languages: languages with fewer than 18,000 training samples, and a four-language high-entropy group. A re-examined prediction is reassigned to a candidate ranked 2 to 5 that has at least 100,000 training samples and a score within 21 natural-log units of the top score.

The constant is -17 for these weights and was -21 for the 2026-08-11 release. It is an absolute target, and the replacement raised every real token by 1.6094 nats, so the two values are not directly comparable. At c = -17 the constant lowers 1,655 of the 1,940 rows and leaves the other 285 as trained. The per-language thresholds were re-estimated on these weights rather than carried across. The group memberships are unchanged: 1,080 languages in the first group, 26 of them with fewer than 200 usable calibration lines and therefore never re-examined, and scoLatn, bjnLatn, argLatn, vlsLatn in the second.

Measured effect of calibration on these weights:

On UDHR and on CommonLID's tag-level macro F1, re-examination also moves some correct low-margin predictions, which lowers those two numbers. On CommonLID it also lowers the number of lines predicted as languages outside the 109-label set from 32,525 to 25,994, which raises macrolanguage-aware accuracy. The evaluations where calibration raises a number are those whose per-language line counts follow a collection's natural imbalance, over a label set that includes under-resourced languages. Pass calibrated=False where the base behavior is wanted.

New languages can be added to this model without retraining anything else (unilid-add-language); the repository README documents the workflow and its three caveats.

License and notices

Apache-2.0. The weights represent aggregated statistical patterns (per-language token probability estimates) derived from the GlotLID-C corpus, which aggregates publicly available sources that may include materials subject to copyright or other rights. The weights do not store or reproduce expressive text content, and no training data is included in this distribution. If you are a rights holder and believe your content was included in the training data and wish to request its exclusion from future versions, contact the authors. Users are responsible for ensuring their use complies with applicable laws and the licensing requirements of any data they process.