Datasetpublic

KIEFERSA/greek-nlu-bench

Greek NLU Benchmark Frozen (v1.0) Greek language-understanding benchmark — 9,751 items / 18 tasks, one split per task. Native-verified and train/eval-disjoint. Built to detect CPT/SFT gains and regressions on Greek, weighted toward morphology-sensitive tasks. Item schema One JSON object per line; identical envelope across tasks: { "id": "el-mcq-000042", "task": "mcq", "language": "el", "version": "1.0", "tags": {"category": "linguistic", "phenomenon":… See the full description on the dataset page: https://huggingface.co/datasets/KIEFERSA/greek-nlu-bench.

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes36downloads

Dataset Card

Greek NLU Benchmark

Frozen (v1.0) Greek language-understanding benchmark — 9,751 items / 18 tasks, one split per task. Native-verified and train/eval-disjoint. Built to detect CPT/SFT gains and regressions on Greek, weighted toward morphology-sensitive tasks.

Item schema

One JSON object per line; identical envelope across tasks:

json

{
  "id": "el-mcq-000042", "task": "mcq", "language": "el", "version": "1.0",
  "tags":       {"category": "linguistic", "phenomenon": "case_agreement", "difficulty": "hard", "domain": "...", "use_case": "...", "split": "dev"},
  "input": { },   "gold": { }
}

input/gold shapes are task-specific (see table). Enums: category ∈ {linguistic, knowledge}; difficulty ∈ {easy, medium, hard}; split ∈ {dev, test} (this release is dev).

Tasks

Split	n	`input` → `gold`	Metric
`acceptability`	820	sentence → 0/1	acc + MCC
`assertion_reason`	450	assertion+reason+4 choices → letter	acc
`cloze_open`	408	text w/ `___` → accepted forms	acc
`cloze_select` ★	331	text w/ `___` + inflected options → index	acc (sliced by phenomenon)
`coreference`	447	text + mention + gender-matched candidates → index	acc
`extractive_qa`	438	context+question → answer spans (`[]` = unanswerable)	EM + token-F1
`idiom_appropriateness`	900	sentence+idiom → 0/1	acc + F1
`idiom_open_cloze`	325	text w/ blanked idiom → accepted	acc
`machine_translation` †	436	source + direction → references	chrF + COMET
`masked_token`	397	text w/ `[MASK]` → accepted	acc
`matching`	441	left[]+right[] → pairs	mean pair-acc (Jaccard)
`mcq`	446	question+4 choices → letter	acc
`nli`	1299	premise+hypothesis → {entail, neutral, contradict}	acc + macro-F1
`paraphrase`	874	sentence1+sentence2 → 0/1	acc + F1
`sentiment`	450	text + label_set → label	acc + macro-F1
`sequence_ordering`	450	shuffled sentences → permutation	Kendall τ
`sts`	450	sentence1+sentence2 → score 0–5	Pearson + Spearman
`wsd`	389	sentence + target + senses → index	acc

★ primary morphology sensor · † separate track — excluded from the NLU macro average

Scoring

—Headline: macro-average of per-task accuracies (MT track reported separately).
—Greek-aware normalization: NFC; accents stripped except where the phenomenon under test is the accent; final sigma folded unless tested.
—Choice tasks support a logprob path (argmax over option continuations) to remove parse noise; generation/regression tasks use greedy decode + parse.
—Primary deliverable is the sliced table (by category / phenomenon / difficulty) and a baseline→candidate diff with regression flags.

python

from datasets import load_dataset
nli = load_dataset("KIEFERSA/greek-nlu-bench", "nli")["dev"]   # one config per task

Reference harness: `greek-bench` · Leaderboard.