overthelex/ua-judge-exam

UA-JudgeExam

11,990 four-option multiple-choice items with official answer keys, from the question bank the Higher Qualification Commission of Judges of Ukraine (Вища кваліфікаційна комісія суддів України, VKKS) publishes for the anonymous written testing of candidates for appellate-court judgeships.

Source: Commission decision of 15 July 2024, No. 221/зп-24 — five documents, 1,672 PDF pages, covering general legal knowledge plus the administrative, commercial, criminal and civil specialisations.

This release accompanies the paper Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks.

Why this release is unusual

Most multiple-choice benchmarks report one number: how often a model picks the key. We also report what a model scores when the question is hidden and only the four options are shown.

On this bank the blind score is far above the 0.250 chance rate — Haiku 4.5 reaches 0.383 pooled over eight option-order permutations — and it is concentrated in a minority of items: 11.8% are answered blind on all eight permutations, 29.3% on none.

Two things it is worth being precise about, because both are easy to overstate:

• —This is not a property of legal exams. Running the identical probe on LEXam's four-choice split returns chance for both Sonnet 4.6 and GPT-5.6 (0.228, upper bound 0.249). In every one of LEXam's 1,655 four-choice items all four options are references to statements in the stem ("i und iii"), median 9 characters against 42 here. The ceiling is sharper than the median: the longest option anywhere in that split is 33 characters, and 59.6% of our options are longer than LEXam's longest, so there is nothing to recognise without the question. Item format decides whether option-only solvability is possible; model capability decides how much is extracted where it is.
• —The capability association is real but loosely estimated. Blind accuracy is a near-constant fraction of full accuracy (0.384–0.538, mean 0.463) across eleven held-out models spanning 0.436–0.958 full-condition accuracy; the two correlate at r = 0.916, but that estimate shares items between the axes. Against an independent capability estimate on disjoint items it is r = 0.813, 95% CI [0.415, 0.950], on eleven points, stable under leave-one-out (0.809–0.854). The five smallest models were added for exactly this reason: on the original seven the interval ran [0.232, 0.976] and barely excluded zero.

Use the gated subset if you need the cleaner set. Read the paper before treating either as leak-free.

Loading

Each file is a separate config; they deliberately have different schemas (the swap variant carries donors, the cross-vendor sample carries _grp), so do not load them together.

from datasets import load_dataset
full  = load_dataset("overthelex/ua-judge-exam", "full")["train"]           # 11,990
gated = load_dataset("overthelex/ua-judge-exam", "gated")["train"]          # 8,128

Files under predictions/ are run artifacts rather than dataset splits and are not exposed as configs; read them directly with huggingface_hub.hf_hub_download.

Files

Item schema

{
  "id": "vkks-2024-civil-0213",
  "set": "civil",
  "question": "Чи може фізична особа, цивільна дієздатність якої обмежена, відмовитися від прийняття спадщини?",
  "options": ["...", "...", "...", "..."],
  "answer_index": 1,
  "answer": "...",
  "n_options": 4,
  "page": 23,
  "source_file": "dodatok_5_2.pdf",
  "question_number": 213,
  "number_inferred": false,
  "status": "active"
}

number_inferred marks the 942 items whose number cell did not extract and was reconstructed; the reconstruction produces zero duplicate numbers across the bank.

Extraction fidelity

Items were extracted with a table parser and verified by a second, independent path that rebuilds rows from the table's ruling lines and cells from word coordinates, sharing no code with the first. Over the full bank the second path confirms the question text in 0.9896 of items, option order in 0.9888, and the answer key in 0.9841. 192 items (1.6%) fail at least one check and are listed in predictions/extraction_verification_failures.json. On a 200-item stratified pilot all three checks agree at 0.995.

How the gated subset was made

Each item is put to Haiku 4.5 with the question withheld, over eight random permutations of option order. Under no leakage each trial is correct with probability 0.25, so eight trials give a binomial reference: P(X >= 5) = 0.027. An item is rejected at five or more hits, and accepted only if at least six of the eight trials returned a parseable letter, so that nothing is certified clean on the strength of two or three responses. Of the 8,137 items at four hits or fewer, nine fail the minimum-trials rule, leaving 8,128.

The gate is not rejected on the lower tail: zero hits out of eight happens to 10.0% of clean items by chance. The model that selects is not a model we report on — every blind figure here comes from models that took no part in selection.

Per-item hit counts were not persisted by the gating run and are therefore not in this release; the distribution over the full bank is 3,517 / 1,410 / 1,196 / 1,047 / 967 / 863 / 827 / 744 / 1,419 for 0 through 8 hits.

Known properties (measured, not assumed)

Blind accuracy on the gated subset, held-out models, with 95% Wilson intervals and the number of parseable responses out of 400. Note that DeepSeek R1 and Pixtral Large fall short of 400 by different amounts in the blind and full conditions, so for those two the two conditions are not scored on identical item sets:

Llama 3.1 8B is set apart and excluded from the correlation: it failed to emit a parseable letter on 49.5% of full-condition calls, so its full-condition figure rests on a self-selected half of the sample. Doubling its budget to 4,096 tokens leaves that at 49.0%, so it is the model and not truncation; its blind figures parse at 96% and are sound. Three of the small models (Gemma 3 12B, Ministral 8B, Llama 3.1 8B) overlap with the small open-source group evaluated by LEXam, which makes the two model sets partially comparable.

Gating removes the leak for nine of these eleven; it does not for Sonnet 4.6 or GPT-5.6.

Measured again on the whole accepted set

The table above rests on a 400-item sample, which turned out to hide two things. Re-running the blind condition on all 8,128 accepted items (105,664 calls, predictions/scenario_a.jsonl) changes what the numbers mean:

• —Subtract each model's own answer-position habit and almost nothing is left. A model that answers A most of the time scores well here because the key sits at A in 29% of accepted items. Scoring each model against what its own pick distribution alone would earn, only two of twelve keep an excess: GPT-5.6 +0.265 and Sonnet 4.6 +0.081. Everything else lands within 0.015 of its habit. Llama 3.1 8B is the cautionary case — 0.292 blind, above every model but the two real leakers, and an excess of +0.005, because it answers A to 92% of items.
• —The gate hands on the blind spots of the model that built it. It keeps the items Haiku 4.5 got wrong, so a model that shares Haiku's preferences inherits them. Agreement with Haiku's blind pick runs 0.42–0.52 where independence would give 0.25, and it tracks the damage: agreement correlates at r = 0.892 with the drop from rejected to accepted items over twelve models — though that figure leans on one point, falling to 0.577 without Llama 3.1 8B.
• —Filtering does not transfer. The gate ran until Haiku 4.5 scored 0.204 blind on what survived. GPT-5.6, which took no part in the selection, still answers 0.515 of those same items with the question hidden. That is the paper's central result, and no repair we tried produced a neutral set.
• —Sample size was doing the concealing. At n=400 the Wilson half-width is 0.045 and nine models read as "statistically at chance". At n=8,128 it is 0.010. Because every model was scored on the same 400-item draw, their errors are correlated and the whole column moved together: nine of thirteen sit lower on the full set, by a mean of 0.010.

Blind accuracy on the full accepted set, with each model's positional habit alongside:

All thirteen models now cover the whole accepted set. Pixtral Large is quota-bound on its Bedrock inference profile — its quota sits on the profile rather than the endpoint, so sharding across five regions did not help and it ran at about 40 calls/min — but it completed: 8,125 of 8,128 items parsed, 3 unparsed.

One warning if you subsample data/ua_judge_exam_gated.jsonl: it is stored in the order of the five source documents, and the documents are the specialisations, so its first 2,000 items are 74% administrative. Stratify rather than slice.

Haiku 4.5 is the gate's own model and Llama 3.1 8B does not read the options; both are set apart and neither enters the correlation.

What we tried in order to remove it

One remedy fails decisively and two are cautionary rather than conclusive, which is worth knowing before anyone repeats them:

• —Swapping distractors for real answers overshoots. Replacing distractors with the correct answers of topically neighbouring items drives blind accuracy from 0.386 [0.321, 0.455] to 0.168 [0.122, 0.226] (n=197, disjoint intervals) — below chance, and just as exploitable by inversion. We found no donor rule that lands on chance.
• —Model-written distractors may leak to the writer's own family. Distractors generated by Sonnet 4.6 score 0.381 [0.296, 0.473] against Sonnet 4.6, versus 0.274 [0.201, 0.363] for the human-written originals of the same 113 items. The intervals overlap, so this is not established — but the design risk is cheap to avoid.
• —No evidence that negation items drive the effect, and the test is underpowered: 0.250 [0.138, 0.411] on 36 negation items against 0.416 [0.343, 0.493] on 161 others. Position bias does not explain the pattern. The direct test is under Robustness below, where the key is forced into each of the four slots in turn: the two effects turn out to be orthogonal, and the model with the largest slot preference is not the one that leaks most.

Robustness

Every number in the paper was re-measured against the ways it could have been an artefact of presentation rather than of the items. All on the 400 gate-accepted items, one factor at a time.

Prompt wording. Five phrasings of the blind prompt (spread 0.033-0.061 within a model) and five of the full prompt (spread 0.005-0.037). The blind-to-full ratio recomputed against each full phrasing spans 0.452-0.464 for Sonnet 4.6 and 0.544-0.547 for GPT-5.6.

Option labels. Latin A)-D), Cyrillic А)-Г) and numeric 1)-4), in both conditions and for all three models. Largest shift 0.027; no model answered with a Latin letter when shown Cyrillic labels. The Latin alphabet over Ukrainian text was not doing the work.

Where the key sits. Forcing the key into each of the four slots in turn gives a test with no modelling assumption: if choice were independent of content, the four accuracies would sum to 1.000 and their mean would be exactly chance. Observed sums are 1.085 (Nova Pro), 1.427 (Sonnet 4.6) and 2.009 (GPT-5.6). The position-free estimates land within 0.022 of the single-order numbers reported throughout, so nothing rests on the printed order. Position bias and leakage turn out to be orthogonal: Sonnet has the largest slot preference in the set (it picks the last slot 0.450 of the time) and leaks moderately, GPT-5.6 has almost none and leaks most.

Position or token? Zheng et al. (ICLR 2024) trace multiple-choice selection bias to token bias over the option IDs themselves, which predicts that the preference should move when the labels change. On these items it mostly does not. Sonnet 4.6 picks the first slot 0.092 of the time under Latin A-D, 0.098 under Cyrillic А-Г and 0.090 under digits 1-4; Nova Pro stays in a similar band (0.180, 0.147, 0.182). What the labels do change is the pull towards the last slot, which for Sonnet runs 0.415, 0.477, 0.310. The bias here is positional first and token-dependent second.

Reasoning at inference time. The model set mixes reasoning and conventional models at vendor defaults, as in LEXam, which cannot be equalised downwards -- GPT-5.6 rejects reasoning_effort on Bedrock. Moving it upwards instead: a step-by-step prompt makes Sonnet 4.6 spend 619 output tokens instead of 4 and moves blind accuracy 0.343 -> 0.388, which a paired McNemar test does not separate from noise (p = 0.139), while the same instruction raises the full condition 0.738 -> 0.805 (p = 0.001). Nova Pro under the same instruction stays at chance (0.255 -> 0.228, p = 0.284). DeepSeek R1 spends 927 output tokens per blind item and scores 0.231, at chance, while GPT-5.6 also reasons and scores 0.524. Reasoning is neither necessary nor sufficient for option-only solvability.

DeepSeek R1 token budget. Re-run at LEXam's 4,096 tokens rather than our 2,048, unparsed responses fall from 13.1% to 1.5% while the estimates barely move (0.253 -> 0.231 blind, 0.665 -> 0.648 full).

Known defects in the source

The Commission's published bank contains a small number of malformed items, which we ship as published rather than silently repairing:

• —vkks-2024-commercial-3000 — the fourth option is blank in the source PDF (last row of the last page of dodatok_3_3.pdf, marked неправильна with no text).
• —Four items carry two identical options: vkks-2024-administrative-2485, vkks-2024-commercial-1177, vkks-2024-commercial-2169, vkks-2024-criminal-0161.

None of these fall in the 400-item gate-accepted set used for the probes, which contains no empty options. Across the full bank, 75 of 47,960 options are shorter than three characters; these are legitimate numeric answers (deadlines, counts), not truncation.

Prompts

Both conditions use a single-turn prompt asking for one letter. Options are presented as A)-D) in the order stored in the file, except inside the gate, where the order is permuted per trial. Decoding is greedy where the provider accepts a temperature, with a 2,048-token output budget so that models which reason before answering are not truncated.

Blind condition (UA-JudgeExam, Ukrainian):

Нижче наведено чотири варіанти відповіді на приховане юридичне питання. Питання вам
не показано. Оберіть той варіант, який найімовірніше є правильним. Відповідайте ЛИШЕ
однією літерою: A, B, C або D.

A) ...
B) ...
C) ...
D) ...

Відповідь:

Full condition (UA-JudgeExam, Ukrainian):

Питання кваліфікаційного тестування для кандидатів на посаду судді. Оберіть єдиний
правильний варіант. Відповідайте ЛИШЕ однією літерою: A, B, C або D.

Питання: {question}
A) ...
B) ...
C) ...
D) ...

Відповідь:

The LEXam comparison uses the same two prompts in English, since that split is German and English rather than Ukrainian:

Below are four answer options to a hidden legal exam question. The question is not shown
to you. Choose the option most likely to be correct. Reply with ONE letter only: A, B, C or D.

Legal exam question. Choose the single correct option. Reply with ONE letter only:
A, B, C or D.

Parsing. Vendors answer in three shapes: a bare letter, an explicit marker (Відповідь: X / Answer: X) followed by prose, or a paragraph of reasoning ending in the letter. The parser tries the explicit marker, then a leading letter, then the last standalone letter. Unparsed responses are excluded from that model's denominator rather than scored as wrong; rates are below 0.5% for every model except DeepSeek R1 (13.1%), which reasons at length and does not always reach a letter within the token budget.

Prompt sensitivity

The blind numbers above use one prompt, so we re-ran the blind condition on the 400 gate-accepted items under five phrasings. Blind accuracy moves very little, the ranking does not change, and a model that sits at chance stays there:

Telling the model the question is legal helps slightly and explains almost nothing: dropping the word costs 0.033 for Sonnet and 0.034 for GPT-5.6. The within-model spread is far smaller than the 0.24 separating Sonnet from GPT-5.6.

The original-prompt row is a re-measurement of the same cell as the cross-vendor table above, on the same items with the same prompt, and gives 0.343 against 0.340. The two runs return identical predictions on 399 of 400 items: that difference is provider non-determinism at temperature 0, not a discrepancy.

Licence and provenance

The bank is published as an annex to a decision (рішення) of the Higher Qualification Commission of Judges. Article 8(1)(3) of the Law of Ukraine On Copyright and Related Rights (No. 2811-IX, in force since 1 January 2023) places outside copyright protection "acts of state authorities ... official documents of a political, legislative, administrative and judicial character (laws, decrees, resolutions, decisions, state standards and the like)".

On that basis the bank is redistributed here with attribution to the issuing decision of 15 July 2024, No. 221/зп-24. Items the Commission itself withdrew (Виключено рішенням) are excluded from this release. This is our reading of the provision, not legal advice; if you intend to rely on it commercially, take your own advice.

The items are examination questions on points of law and contain no personal data.

predictions/lexam_four_choice_blind.jsonl contains our model outputs on LEXam items; the LEXam dataset itself is not redistributed here and remains under its own licence.

Intended use and limits

Built for evaluating and auditing legal reasoning benchmarks, not for judicial decision-making or for preparing candidates. The strongest model tested reaches 0.958 on the gated subset, so for frontier models this bank is close to saturated; its discriminative value lies below that. There is no human blind baseline.

Citation

@misc{ovcharov2026uajudgeexam,
  title  = {Gated Against One Model, Open to the Next: Option-Only
            Solvability in Legal
            Multiple-Choice Benchmarks},
  author = {Ovcharov, Volodymyr},
  year   = {2026}
}