Jel1f1sh/tw-legal-benchmark-v2

Taiwan Legal Benchmark v2

A multiple-choice benchmark for evaluating LLMs on Taiwan law in Traditional Chinese, built from 15 years (2012–2026) of national examinations published by the Ministry of Examination (考選部).

Supersedes tw-legal-benchmark-v1 (209 questions) with 17,002 deduplicated questions across 15 legal domains.

Overview

Two tiers

The benchmark is explicitly stratified by how law-dense the source paper is, because Taiwan's national examinations differ enormously in legal depth.

Filter with tier == 1 for the hardest, most purely legal subset.

Tier 1 is near-complete coverage of its population. Across 2012–2026 the Ministry of Examination published only 177 core-legal multiple-choice papers in total; this dataset draws on 172 of them (97%). Taiwan's senior legal subjects are overwhelmingly examined by essay, not multiple choice, so the pool of serious legal MCQs is small by construction — not by sampling.

Subject Distribution

Examination Levels

Fields

Usage

Twinkle Eval

uv run scripts/make_eval_config.py \
    --model my-model --base-url http://localhost:8000/v1 \
    --bench lianghsun/tw-legal-benchmark-v2 --out eval/
twinkle-eval --config eval/config.yaml --export json csv

If a model scores near zero, check the 無法解析 rate before believing the number. A stock instruct model answering 最終答案：C scores ~5% under the box method and ~30% under pattern — same model, same questions. Match the extraction method to the model.

datasets

from datasets import load_dataset
ds = load_dataset("lianghsun/tw-legal-benchmark-v2", split="train")
core = ds.filter(lambda r: r["tier"] == 1)      # hardest subset

Construction

1. 1.Index — the official MOEX open-data index (wHandExamQandA_CSV.ashx, 66,582 rows), filtered to statute-grounded multiple-choice papers: 1,040 papers.
2. 2.Retrieval — 2,080 question and answer-key PDFs fetched (rate-limited, checksummed, 0 failures).
3. 3.Parsing — 986/1,040 papers parsed completely; 56,479 raw questions.
4. 4.Filtering and deduplication — see below.

Parsing hazards handled

These silently corrupt naive extraction, so they are documented here:

• —Option markers live in the EUDC Private Use Area (U+E18C–U+E190 = A–E) rather than as literal letters — and one legacy layout uses plain 1. / A. instead.
• —Text mixes CJK Compatibility Ideographs (e.g. 不 U+F967). Normalised with NFC — not NFKC, which rewrites 「，」to an ASCII comma and breaks matching against statute text.
• —Papers mix single-select and multi-select sections; multi-select answers are multi-letter tokens (ABDE) that a per-character regex shreds, misaligning every later row.
• —Answer sheets appear in several layouts: 01 vs 第1題 numbering, 題號 vs 題序 labels, half-width vs full-width answer letters (Ａ, ＃).
• —Variable spacing after question numbers (one paper year uses a single space).
• —Nine 2014 answer sheets carry an internally inconsistent official range (「單選題數：70題(第1~69題)」 overlapping 複選 61~70).

Deduplication

The same paper is republished under several examination categories (司法官 / 律師 / 司法官及律師, and equivalents at other grades). Before deduplication there are 56,095 single-answer items; after collapsing identical items 17,002 remain. All duplicate groups were verified to carry identical answer keys (0 conflicts). n_source_papers records the multiplicity.

A benchmark built from this source without deduplication is inflated roughly 2–3×, and accuracy is skewed toward the republished subset.

Excluded

Known Properties

• —Answer-position bias in the source: A=3,521, B=4,334, C=4,639, D=4,509. Use shuffle_options: true (Twinkle Eval default).
• —Contamination risk: these are public past papers, widely discussed online and very likely present in pretraining corpora. Absolute scores should be treated with caution; the benchmark is most useful for relative comparison between models.
• —Whitespace normalisation removes spaces inside stems, which is correct for Chinese and would be wrong for Latin text — the English sections are excluded for this reason too.
• —Item difficulty is set by the examination committee, not by us.

Licensing

Questions and answer keys are official publications of the Ministry of Examination, Republic of China (Taiwan), released for public access. The packaging, parsing, deduplication, and metadata in this repository are provided under Apache-2.0, following the convention of v1. Users intending redistribution should confirm the current terms published by 考選部.

Citation

@misc{twlegalbenchv2,
  title  = {Taiwan Legal Benchmark v2},
  author = {Huang, Liang-Hsun},
  year   = {2026},
  url    = {https://huggingface.co/datasets/lianghsun/tw-legal-benchmark-v2}
}