overthelex/ua-judge-exam
UA-JudgeExam 11,990 four-option multiple-choice items with official answer keys, from the question bank the Higher Qualification Commission of Judges of Ukraine (Вища кваліфікаційна комісія суддів України, VKKS) publishes for the anonymous written testing of candidates for appellate-court judgeships. Source: Commission decision of 15 July 2024, No. 221/зп-24 — five documents, 1,672 PDF pages, covering general legal knowledge plus the administrative, commercial, criminal and… See the full description on the dataset page: https://huggingface.co/datasets/overthelex/ua-judge-exam.
Sync the card's correlations and ratio to the completed full-set run
Complete the full-scale blind run: all 13 models on all 8,128 items
Full-scale blind run: 13 models on all 8,128 gate-accepted items
Solvability-profile verification scripts
Solvability-profile verification scripts
Solvability-profile verification scripts
Solvability-profile verification scripts
Corpus and extraction-fidelity verification scripts
Corpus and extraction-fidelity verification scripts
Position versus token bias; lexical-search verification
Position versus token bias; lexical-search verification
Position versus token bias; lexical-search verification
Update consistency check for the extended model set
Extend the model set downwards with five small models
Extend the model set downwards with five small models
Extend the model set downwards with five small models
Extend the model set downwards with five small models
Extend the model set downwards with five small models
Extend the model set downwards with five small models
Extend the model set downwards with five small models
Add paper/card consistency check
Correct the LEXam count, document the gate rule, add figure inputs
Correct the LEXam count, document the gate rule, add figure inputs
Correct the LEXam count, document the gate rule, add figure inputs
Add table verification and figure scripts
Add table verification and figure scripts
Add table verification and figure scripts
Add table verification and figure scripts
Add table verification and figure scripts
Add table verification and figure scripts
Add table verification and figure scripts
Add presentation, position and reasoning robustness runs
Add presentation, position and reasoning robustness runs
Add presentation, position and reasoning robustness runs
Add presentation, position and reasoning robustness runs
Add presentation, position and reasoning robustness runs
Add presentation, position and reasoning robustness runs
Add presentation, position and reasoning robustness runs
Add presentation, position and reasoning robustness runs
Add presentation, position and reasoning robustness runs
Add presentation, position and reasoning robustness runs
Add prompt-sensitivity ablation: 6,000 blind predictions under five phrasings, plus results in the card
Report confidence intervals on the remedies: two of three are underpowered, not findings
Add the exact prompts and answer-parsing rules to the card
Sync card with paper: LEXam comparison, position-bias control, correlation caveat, new title; add LEXam predictions
Add per-file configs: files have different schemas, viewer was globbing them into one split
Fix licence citation: current law 2811-IX art. 8(1)(3), was linking repealed 3792-XII
UA-JudgeExam: 11,990 items with official keys, gated subset, cross-vendor predictions, code
initial commit