kamaalg/azerbaijani-eval-benchmarks

Azerbaijani evaluation benchmarks (v0)

Small, reproducible Azerbaijani benchmarks for evaluating base language models, part of an open Azerbaijani LLM stack. Built by build_benchmarks.py (rerun to regenerate deterministically).

mmlu_az.jsonl

MMLU-style 4-way multiple choice, scored by answer log-likelihood (works on a base model — no instruction tuning needed). See evals/mini_mmlu_az.py. Each item has a subject; the correct-answer position is shuffled (seeded) so it is roughly uniform across A/B/C/D.

Subjects (≈12–14 each): az_history, az_geography, az_literature, az_language (grammar/vocab), world_knowledge, science, math (computed), civics_law.

{"question": "Azərbaycanın paytaxtı hansı şəhərdir?", "choices": ["Gəncə","Bakı","Sumqayıt","Şəki"], "answer": 1, "subject": "az_geography"}

ner_az.jsonl

CoNLL-style BIO named-entity recognition — entity types PER, LOC, ORG — scored with strict entity-level F1 (evals/ner.py; run with no --pred to self-check the gold at 100%). Entities: LOC 39, ORG 9, PER 16 across 42 native Azerbaijani sentences about real, well-known entities.

{"tokens": ["Bakı","Azərbaycanın","paytaxtıdır","."], "tags": ["B-LOC","B-LOC","O","O"]}

Provenance & scope

• —Model-authored v0, restricted to well-established, canonical facts (capitals, dates every Azerbaijani learns in school, arithmetic, basic science). Math answers are computed; the builder asserts 4 unique choices, valid answer indices, and well-formed BIO.
• —Language: Latin-script Modern (North) Azerbaijani; the i/İ/ı/I distinction is preserved.

Limitations (honest)

• —Not a professionally annotated set. It is a v0 for smoke-testing and relative comparison, pending native-speaker validation. Treat absolute scores with caution.
• —Canonical-facts-only ⇒ it under-tests nuance, reasoning depth, and long-tail knowledge.
• —Small: 102 MC / 42 NER. Enough to rank models coarsely and catch regressions, not to certify quality.
• —The NER set covers famous entities and clean prose — real-world NER (noisy web text, rare names) is harder than this suggests.

How to expand / contribute

Add items to the MC / NER lists in build_benchmarks.py and rerun. MC items are authored as (subject, question, correct_text, [distractors]) — you specify the correct answer by content, never an index, so it cannot be mislabeled. NER sentences use a compact (text, tag) span spec that expands to well-formed BIO automatically. Native-speaker review of the existing items is very welcome.

License

CC-BY-4.0.