khyentsevision/tibetan-ai-leaderboard

AI Benchmarks for Tibetan (བོད་སྐད་)

A leaderboard aggregating AI model benchmark results for Tibetan (bo, Sino-Tibetan) across seven independent evaluation efforts: LanguageBench (live translation and classification), DharmaBench (Buddhist literary tasks), TLUE (general knowledge and safety), MiLiC-Eval (broad LLM evaluation), TNCC (news classification), WCM (cross-lingual transfer), and TibetanQA (reading comprehension).

Benchmarks

LanguageBench

Source: fair-forward/languagebench, filtered to bcp_47 == "bo". Data: fair-forward/evals-for-every-language-results

LanguageBench evaluates a broad set of AI models on translation and classification tasks for low-resource languages. Results are fetched live from the upstream HuggingFace dataset at server startup and are updated whenever the dataset is refreshed.

Tasks (Tibetan):

Results are based on a sample of 10 sentences per task. Bootstrap 95% confidence intervals are shown as error bars — treat small differences between models with caution. The overall score is the simple mean across the three tasks.

Translation Quality Metrics

Translation Quality Metric Validity for Tibetan–English — KVP internal study (2026).

• —[LLMJudgeReport.md](LLMJudgeReport.md) — the twelve-judge GEMBA-DA results.
• —[MetricCorrelationResults.md](MetricCorrelationResults.md) — results for all three families.
• —Scripts: scripts/compute_metrics.py, scripts/compute_ref_free_metrics.py, scripts/compute_llm_metrics.py (see scripts/README.md).

DharmaBench

Reference: Hashiloni et al. (2025). DharmaBench: Evaluating Language Models on Buddhist Texts in Sanskrit and Tibetan. IJCNLP-AACL 2025.

DharmaBench evaluates language models on Buddhist literary and scriptural texts. This leaderboard shows Tibetan results only (Sanskrit results are excluded). The benchmark tests a range of literary and linguistic skills specific to classical Tibetan Buddhist literature.

Tasks:

SDT and QUDT are span detection tasks requiring processing of long documents. Encoder-only models (mBERT, XLM-RoBERTa, etc.) are fine-tuned for classification and cannot perform these span tasks due to the 512-token context limit; their SDT and QUDT cells are shown as —. The Avg. column and ranking exclude encoder models, since they are not directly comparable to generative models.

Results are static (embedded from the paper); they are not updated automatically.

TLUE

Reference: Gao et al. (2025). TLUE: A Tibetan Language Understanding Evaluation Benchmark. EMNLP 2025.

TLUE assesses AI models on Tibetan language understanding across two sub-benchmarks covering knowledge breadth and safety alignment.

Ti-MMLU — 11,528 multiple-choice questions across 67 subjects in five domains:

Random baseline: 25.00% (four-choice questions).

Ti-SafetyBench — 11,435 questions across seven safety categories:

Random baseline: 36.7%.

Metric: CAA (Concern All Answer) Accuracy ×100% — an upper-bound accuracy estimate that credits a model if the correct answer appears anywhere in its response.

Results are static (embedded from the paper); they are not updated automatically.

MiLiC-Eval

Reference: (2025). MiLiC-Eval: Benchmarking Multilingual LLMs for China's Minority Languages. ACL 2025 Findings. arXiv 2503.01150. Dataset: pkupie/milic-eval · GitHub

MiLiC-Eval evaluates general-purpose LLMs on Tibetan, Uyghur, Kazakh, and Mongolian across 24,000 instances and 11 tasks. Only Tibetan results are shown here. No Tibetan-specific models were included in the original evaluation.

Tasks (Tibetan):

Results are static (embedded from the paper); they are not updated automatically.

TNCC

Original corpus: Qun et al. (2017). End-to-End Neural Text Classification for Tibetan. Dataset: UTibetNLP/tibetan_news_classification

TNCC is a Tibetan news classification corpus collected from China Tibet Online, with 12 categories (politics, economics, education, tourism, environment, language, literature, religion, arts, medicine, customs, instruments). Two sub-tasks: full-article classification (TNCC-document) and headline-only classification (TNCC-title).

Results are compiled from multiple papers (2022–2025) that use TNCC as a downstream evaluation benchmark. The primary metric is Macro-F1 on TNCC-document. Note that different papers report different metrics (some accuracy, some weighted F1); inconsistencies are noted per model.

WCM

Reference: Yang et al. (2022). CINO: A Chinese Minority Pre-trained Language Model. COLING 2022. ACL Anthology

WCM (Wiki-Chinese-Minority) is a Wikipedia-based topic classification benchmark covering 7 languages with 10 classes and ~63,000 examples. The evaluation shown is zero-shot cross-lingual transfer: models are trained on Standard Chinese and tested on Tibetan, measuring how well multilingual representations generalize to a low-resource language with virtually no overlap in script or typology.

Metric: Macro-F1 (%).

Results are static (embedded from the paper); they are not updated automatically.

TibetanQA

Reference: Sun et al. (2021). Construction of High-quality Tibetan Dataset for Machine Reading Comprehension. CCL 2021. GitHub

TibetanQA is the first large-scale extractive MRC dataset for Tibetan. Given a passage from the Tibetan Encyclopedia, models extract a text span to answer a question. 20,000 QA pairs, 1,513 articles, 12 topics. All pairs constructed and verified by native Tibetan speakers.

A second version, TibetanQA 2.0 (Dan & Sun, 2024, Data Intelligence), extends the dataset with 1,347 unanswerable questions (12,054 total), but has no published model results.

Metric: F1 and Exact Match (EM), both 0–100. Only one public submission (Ti-Reader) has been recorded on the leaderboard.

Results are static (embedded from the paper); they are not updated automatically.

Architecture

The leaderboard is a single Docker container that serves both the backend API and the compiled React frontend.

tibetan-leaderboard/
├── Dockerfile              # Multi-stage build: Node → Python
├── pyproject.toml          # Python dependencies (FastAPI, uvicorn, pandas, etc.)
├── evals/
│   └── backend.py          # FastAPI app + data processing
├── data/
│   └── datasets.json       # Dataset metadata for the DatasetTable
└── frontend/
    ├── package.json
    └── src/
        ├── App.js           # Root component, TabView layout, dialogs
        ├── components/
        │   ├── ModelTable.js        # LanguageBench model ranking table
        │   ├── DatasetTable.js      # Dataset metadata table
        │   ├── ScoreField.js        # Colored bar + CI error bar cell renderer
        │   ├── Medal.js             # Gold/silver/bronze rank medals
        │   ├── ScoreColumns.js      # Column definitions for LanguageBench metrics
        │   ├── HistoryPlot.js       # Score-over-time chart
        │   ├── LicenseHistoryPlot.js # License breakdown chart
        │   ├── CostPlot.js          # Cost vs. performance chart
        │   ├── DharmaBenchTab.js    # DharmaBench results table
        │   ├── TLUETab.js           # TLUE results table (Ti-MMLU / Ti-SafetyBench)
        │   ├── MiLiCEvalTab.js      # MiLiC-Eval results table (11 tasks)
        │   ├── TNCCTab.js           # TNCC news classification results
        │   ├── WCMTab.js            # WCM cross-lingual transfer results
        │   └── TibetanQATab.js      # TibetanQA reading comprehension results
        └── data/
            ├── dharmabench.js       # Static DharmaBench paper results
            ├── tlue.js              # Static TLUE paper results
            ├── milic_eval.js        # Static MiLiC-Eval paper results (Tibetan)
            ├── tncc.js              # Static TNCC results (compiled from multiple papers)
            ├── wcm.js               # Static WCM cross-lingual transfer results
            └── tibetanqa.js         # Static TibetanQA leaderboard results

Build pipeline (Dockerfile)

Stage 1 — Node 20 Alpine: Installs npm dependencies and runs npm run build (Create React App), producing a content-hashed static bundle in frontend/build/.

Stage 2 — Python 3.12 (uv): Installs Python dependencies with uv sync. Copies the compiled frontend build from stage 1. Runs evals/backend.py with uvicorn on port 8000.

Backend (evals/backend.py)

A FastAPI application with two responsibilities:

1. 1.`GET /api/data` — loads evaluation results from the fair-forward/evals-for-every-language-results HuggingFace dataset at startup (via the datasets library), filters to bcp_47 == "bo", computes per-model averages and bootstrap 95% confidence intervals, and returns a JSON payload containing model_table and dataset_table.

1. 1.Static file serving — mounts the React build directory at /, so all non-API routes serve the SPA.

Confidence intervals are computed with 1,000 bootstrap resamples over the per-sentence scores in the detailed results dataset and cached with joblib.Memory to avoid recomputation on repeated requests.

Frontend (frontend/src/)

A React 18 SPA built with Create React App, using PrimeReact as the component library.

The UI is organized as a TabView with eight tabs:

• —LanguageBench — fetches /api/data on mount and renders a sortable, filterable ModelTable (with provider, model type, size, and cost filters), a DatasetTable, and a Carousel of three Plotly charts (score history, license breakdown, cost vs. performance). A fullscreen dialog is available for the charts.
• —Translation Quality Metrics — MetricCorrelationTab.js, with a SelectButton for All / Reference-based / Reference-free / LLM. Correlation values are embedded constants (REF_BASED, REF_FREE, LLM_JUDGE), transcribed from the scripts' output. The LLM sub-tab renders a report summary, the LlmJudgePlot cost-vs-correlation figure, and a ColumnGroup table pairing each judge's with-reference and no-reference correlations alongside measured cost.
• —DharmaBench — static DataTable from data/dharmabench.js. Seven Tibetan Buddhist literary tasks. Scores use the shared ScoreField bar renderer (values divided by 100 for normalization) and Medal for rank. Error bars show ±1 std dev from multiple runs.
• —TLUE — two static DataTables (Ti-MMLU and Ti-SafetyBench) from data/tlue.js, switchable via a SelectButton. Same ScoreField and Medal styling.
• —MiLiC-Eval — static DataTable from data/milic_eval.js. 10 models × 11 task columns (Tibetan only). Same styling.
• —TNCC — static DataTable from data/tncc.js. Results compiled from multiple papers (2022–2025) using TNCC as a benchmark. Ranked by document Macro-F1.
• —WCM — static DataTable from data/wcm.js. Zero-shot cross-lingual transfer Macro-F1 on Tibetan Wikipedia classification. 4 models.
• —TibetanQA — static DataTable from data/tibetanqa.js. Extractive MRC, F1 and Exact Match. Human performance shown as upper bound.

All static tab data is embedded directly in the frontend bundle (no API call required) since it is compiled from published papers and does not change.

Note on cross-benchmark comparison

Results across the seven tabs come from different evaluation setups, model versions, prompting strategies, and time periods. They should not be directly compared across benchmarks. Within each benchmark, medal rankings and colored bar scores indicate the best-performing model per column.