yukobayashi500/vntl-leaderboard-2026

VNTL Leaderboard — 2026 Edition

A revival of lmg-anon's vntl-leaderboard (Japanese→English visual novel translation), which stopped updating in January 2025. This edition keeps all 87 original entries on the exact same footing and adds 4 current models, for 91 entries total.

Headline results (new models, evaluated 2026-09-07)

Two findings stand out:

• —shisa-v2-mistral-nemo-12b is the strongest newly-tested local model, beating every local model except qwen-2.5-72b, vntl-gemma2-27b and qwen-2.5-32b.
• —Sugoi-14B-Ultra scores below vntl-llama3-8b-v2 (0.6952, rank 19), a fine-tune that is 20 months older and 6B parameters smaller. Caveat: Sugoi was evaluated at Q4KM (the quant its authors distribute) while vntl-llama3-8b-v2 was evaluated at Q80, so part of the gap may be quantization. Even so, Q4K_M is how most people actually run it.

New in this edition: per-work stability

Every leaderboard in this space reports one number per model — here, the mean over two visual novels (Senren\*Banka and Mashiro-iro Symphony). That single number hides something big.

*One system ranks #1 of 93 on Senren\Banka and #91 of 93 on Mashiro-iro Symphony.**

stability.jsonl splits every score back into its two works and reports how far apart they are.

How it is measured

For each model, gap = accuracy(Senren) − accuracy(Mashiro), using the same leaderboard aggregation (per-line accuracy ≤ 0.5 counted as 0, then mean).

Every model scores higher on Senren — the cohort mean gap is +0.1007. So a positive gap is normal and is not by itself evidence of anything. The question is how far a model sits from that cohort baseline. Each model's gap gets a bootstrap 95% CI (2,000 resamples, each work resampled independently, seed 20260915), and a model is flagged work_dependent only when its CI excludes the cohort mean gap — i.e. its own measurement error cannot explain the distance.

Honest limit, stated up front: with 128 lines per work, roughly half the observed spread in gaps is measurement noise (observed SD 0.0527 vs noise SD 0.0372). That is why only 7 of 93 models are flagged, and why you should not read small rank_delta values as real. Mid-pack rank swings of ±20 are well within noise.

The pattern: off-the-shelf MT services, not LLMs

Difference +0.172, permutation test (20,000 shuffles) p = 0.00005.

All four sentence-level MT services in the data are work-dependent; almost no LLM is. This looks structural rather than suspicious: sentence-level MT has no access to surrounding context, and Mashiro-iro Symphony's excerpt leans more on long narration and context-carrying lines than Senren\Banka's does. We are not claiming training-data contamination — verbatim-match rates for these systems are not* outliers (z ≈ +1.1), so memorised reference text does not explain it. Separating "genre/style weakness" from "data overlap" would need a third work, which this edition does not have.

The seven flagged models

The last two are flagged in the opposite direction: they are unusually insensitive to the change of work. qwq-preview is 59th on Senren and 7th on Mashiro.

Note on naming. _Sugoi Translator is the classic sentence-level Sugoi translator, one of four off-the-shelf services the original benchmark ran but never listed in its ranking table. It is a different system from sugoitoolkit/Sugoi-14B-Ultra (an LLM, rank 31 in the leaderboard above, gap +0.082, not flagged). The four services are included in stability.jsonl because the per-line data exists; they are not part of the 91-entry ranking.

Note on ranks. All ranks in this section and in stability.jsonl are positions within the 93-row stability cohort, which drops 2 ranked entries and adds the 4 MT services. They therefore differ slightly from rank in leaderboard.jsonl (91 entries). The accuracy column is identical in both files.

What this is useful for

If you are picking a translator for your own text, the leaderboard mean is a weak predictor for the MT-service class: its score on one work told you very little about the next. For LLMs the mean held up much better. Either way, 128 lines × 2 works is a small sample — treat stability.jsonl as a warning flag, not a verdict.

Coverage

93 rows = 89 of the 91 ranked entries + the 4 off-the-shelf MT services. Two ranked entries are excluded because the upstream per-work data is unusable: x-ai/grok-beta (no per-work result files) and lmg-anon/vntl-13b-v0.2 (its Mashiro file is empty).

Why these numbers are comparable to the original leaderboard

The original evaluation data was never published, so we reconstructed the pipeline and verified it against the original's own published results in three independent ways:

1. 1.Scoring: re-scored an original results file (128 lines) with our scorer (all-mpnet-base-v2 cosine similarity + chrF + BLEU, same cleaning rules). Maximum accuracy difference: 0.000001.
2. 2.Generation: re-evaluated vntl-gemma2-2b#Q8_0 end-to-end with llama.cpp (temperature 0, greedy, stop ["\n"]). Mean accuracy over all 256 samples came out within 0.0003 of the original's published raw scores.
3. 3.Aggregation: the leaderboard score (per-sample accuracy ≤ 0.5 counted as 0, then mean; CI = 1.96×SEM) reproduces the original's published values bit-exactly.

The evaluation set is the same as the original: 128 lines from Senren*Banka plus 128 lines from Mashiro-iro Symphony (current dataset revision), with identical few-shot prompts, reconstructed from the prompt/expected fields the original leaderboard published in its per-model result files.

New models were generated on a Colab T4 with llama.cpp (CUDA, -ngl 99), greedy decoding, n_predict 150, stop ["\n"] — identical parameters to the original harness.

Full leaderboard (91 entries)

<details> <summary>Click to expand the full table</summary>

</details>

Files

• —leaderboard.jsonl — all 91 entries: model, quant, accuracy (leaderboard formula), 95% ci (±), chrf_mean, rank. New 2026 entries carry "evaluated": "2026-09-07".
• —stability.jsonl — 93 rows, per-work breakdown: acc_senren, acc_mashiro, rank_senren, rank_mashiro, rank_delta, gap, gap_ci_low/gap_ci_high (bootstrap 95% CI), excess_vs_cohort, gap_z (leave-one-out), work_dependent, system_type (LLM / MT service), source. accuracy matches leaderboard.jsonl; rank_combined is the position within this 93-row cohort and differs from rank there.
• —results/ — for the 4 new models: raw generations ({id, expected, generated}) and per-sample scores (accuracy, bleu, chrf), one file per model×dataset, plus summary.json (plain mean accuracy over 256 samples, before the leaderboard formula).

Credits

All credit for the benchmark design, the original 87 evaluations, and the scoring method goes to lmg-anon (vntl-benchmark, vntl-leaderboard). This edition only reproduces the method and extends the model list.

Reference translations belong to the respective visual novels' rights holders. If you are a rights holder and want an excerpt removed, open a discussion here and it will be taken down promptly.