lab-ii/YakutEval

YakutEval — offline LLM leaderboard for Yakut (Sakha)

A reproducible benchmark for evaluating LLMs on Yakut (Sakha, `sah`), a low-resource Turkic language. The Space does not run models — you evaluate your model locally and submit the resulting scores JSON.

Tasks (all verifiable)

• —math — Yakut math word problems, checked by the final number.
• —physics — Yakut physics problems (kinematics, force, work), numeric answer.
• —tools — Yakut tool-calling: query → correct tool + arguments.
• —cloze — predict a masked content word in a Yakut sentence.
• —grammar — MC on cases / vowel harmony / verb forms.
• —vocab — MC on word meaning / synonyms (language knowledge).
• —fluency — share of Yakut Cyrillic/markers in free generation.
• —ppl — perplexity on held-out Yakut text (lower is better).

overall = mean of the accuracy tasks (math, physics, tools, cloze, grammar, vocab, fluency).

Submit your model

The benchmark and scorer ship inside this repo, so you only need a GPU to run your own model:

git clone https://huggingface.co/spaces/lab-ii/YakutEval
cd YakutEval
pip install torch transformers pyarrow
python yakuteval/run_eval.py --model <your_hf_id> --name <short_name>

This writes yakuteval/results/<short_name>.json. Open the «Засабмитить результаты» tab, paste that JSON, and Submit — the leaderboard updates.

Scoring is fully verifiable; math/physics gold are by construction, the MC tasks (grammar/vocab) are LLM-authored + self-consistency-filtered (not human-verified).