LocalDoc/YOXLA-Benchmark

YOXLA Benchmark

1443 frozen examples for evaluating large language models in Azerbaijani, across four blocks and eleven tasks.

Run with the YOXLA framework:

pip install "yoxla[api]"
yoxla run --provider openrouter --model <model> --block all

Or load a config directly:

from datasets import load_dataset

data = load_dataset("LocalDoc/YOXLA-Benchmark", "rag_selection_v1")["test"]

What makes it different

Every answer space is closed. A label, a number, or a span quoted from a passage the model was given. Nothing is scored by a judge model, nothing depends on deciding whether two spellings of a name mean the same thing, and every score can be reproduced from stored output.

That decision cost the benchmark two tasks. An orthography block that scored whether a model writes Azerbaijani correctly, and an open-answer knowledge task, were both built and both dropped: sixty-two of the knowledge set's hundred and fifty answers were multi-word entities — honorific titles no two people would word alike, names with up to seven equally correct forms — and every scoring dispute found in review came from that group.

No model decides a label. Generators write passages, sentences and candidate options; the gold comes from Wikidata, from a named grammatical rule, from a dictionary, or from the construction of the item itself. A generation that does not match its source is rejected rather than relabelled.

The distractors are measured, not assumed. Every set with a closed answer space is run past strategies that read nothing — pick the most famous option, pick the passage sharing the most words with the question, always answer the first one — and the build fails if any of them beats its floor. Where a cue cannot be removed, the floor it leaves is computed from the finished data and published with the task rather than hidden.

Blocks

Understanding — 600

The two span tasks are scored on character offsets rather than on strings: the quote is located in the passage and its range compared with the gold's. The gold answer occurs exactly once in its own context in all 250 answerable examples, which is what makes the position unambiguous.

Language — 400

Both show two sentences differing in one word and ask which one is Azerbaijani. Position is balanced inside every breakdown, so answering "A" throughout scores 50.

minimal_pairs_v1 corrupts a named rule — vowel harmony, the question particle, the definite accusative, case government and four more — so the label follows from the rule rather than from an opinion.

az_tr_interference_v1 replaces an Azerbaijani word with a Turkish one:

A) Uşaqlar məktəbdə çox gözəl danışmaq öyrənirlər.
B) Uşaqlar məktəbdə çox gözəl konuşmak öyrənirlər.

Three lookups in two independent sources decide every pair: the Azerbaijani word is in a hunspell dictionary and occurs at least 200 times in Azerbaijani Wikipedia, the Turkish word is in a Turkish dictionary and absent from the Azerbaijani one, and the Azerbaijani form outnumbers the Turkish one at least 50:1 in the corpus. Read interference_type beside the score: a cognate pair (kitab / kitap) asks which spelling Azerbaijani uses, a distinct_lexeme pair (danışmaq / konuşmak) asks which word it uses at all, and only the second is beyond a model that merely spells correctly.

Knowledge — 150

Facts about Azerbaijan harvested from Wikidata across seven categories. A fact Wikidata answers more than one way is dropped at build time.

Three cues are closed at build time so that recognition is not free: distractors come from the same relation, so the wrong kind of thing cannot be eliminated; the answer is not the most-linked option, so "pick the famous one" fails; a year distractor sits within a decade of the answer. Read modal_answer_share beside the score — a model that does not know tends to return the same number every time, and accuracy alone does not show it.

RAG — 293

The two halves of a retrieval system, measured apart: selection asks whether the right passage was picked up, verification asks what the model does with a passage once it has one.

rag_verification_v1 gives a passage and one claim. Its four claim types are split out by claim_type, and the one worth reading is counterfactual: the passage states an altered value — a year moved, a district changed — and the claim states the real one. A model answering TƏSDİQ has read its own memory instead of the text, which is the failure that makes a retrieval system quietly wrong. Absence is decidable here because each passage is written from a known list of three facts, so anything outside that list is provably not stated.

rag_selection_v1 gives a question and six short passages, about seventy words in all. The negatives are the measurement: passages about the same subject but a different aspect, passages about the same aspect but a different subject, and passages whose value is of the same kind as the answer. One item in seven has its answer in none of them — per_item_type_accuracy splits those out, and a system that always returns its best guess fails there and nowhere else.

One caveat specific to this task. The word-overlap cue cannot be driven to chance: the gold is the only passage carrying both the subject and the relation, and a negative carrying both would be a passage stating the answer. The gold is held to a tie with the other passages about its subject instead, which leaves a floor near 0.35 for "pick the passage sharing the most words" — against a chance of 0.14 and against floors at chance everywhere else in this benchmark. Read the score with that in mind.

Sources and how the gold was made

The dictionaries and the frequency table were used as lookups during construction and are not redistributed here. Each is trusted in one direction only: presence in the dictionary proves a form is Azerbaijani, and absence proves nothing, because its verb paradigms have holes.

Generator models wrote passages, sentences and questions. They never decided a label, and no model output was accepted without passing the deterministic check for its task.

What this benchmark does not measure

Production. No task asks a model to write Azerbaijani and scores what it wrote. The same model recognised correct grammar at 90.5 and, in the same run, scored 57.7 on writing it — the two are not interchangeable, and only the first number exists here.

Recall. Knowledge asks a model to pick a fact out of twenty candidates, never to produce it. The interference task asks it to recognise a Turkish word, not to avoid writing one.

Anything above sentence level. No discourse, no long context, no multi-turn behaviour.

Scoring these would need a judge model or human annotation. A judge would put one model's Azerbaijani inside the loop that measures Azerbaijani, and would end reproducibility.

Versioning

A task id is frozen from its first publication. A substantial change means a new id — nli_v2 — never an edit in place, and the superseded task stays runnable so an older run can be reproduced. Pin a revision for a comparable run:

yoxla run --provider openai --model <model> --block all --revision <commit>

The framework bundles a manifest of row counts and content fingerprints per block and warns when the data here has changed since a run was scored.

Licence

Data: CC BY 4.0. Framework: Apache-2.0.