permitt/superglue-mne

BalkanBench SuperGLUE - Montenegrin

Part of [BalkanBench](https://balkanbench.com) - the open, reproducible benchmark for language models across Serbian, Croatian, Montenegrin, and Bosnian (BCMS). Live leaderboard at <https://balkanbench.com/leaderboard>. Background: Release of BalkanBench - the vision behind it (Medium, 2026-04-27).

This is the Montenegrin SuperGLUE released preview track of BalkanBench v0.1, covering the same 5 ranked tasks as the Croatian preview. Same scoring contract as Serbian (the official frozen track), same private test labels, no WSC adaptation yet.

What's inside

5 task configs, full train + validation + public test splits (train and validation labelled here; the test split lives only in the gated sibling repo, see below):

Total: 65,049 items. WSC and the AX-b/AX-g diagnostics are Serbian-only in v0.1.

Test split is gated

The test split for every config lives only in the gated sibling repo (both inputs and labels): `permitt/superglue-mne-private`, and are accessed only by the official scoring pipeline. To get a leaderboard score, generate predictions on the public test inputs with balkanbench predict, then submit through the submission flow.

Loading

from datasets import load_dataset

boolq = load_dataset("permitt/superglue-mne", "boolq")
print(boolq["train"][0])

# All ranked tasks
for cfg in ["boolq", "cb", "copa", "rte", "multirc"]:
    ds = load_dataset("permitt/superglue-mne", cfg)
    print(cfg, {sp: len(ds[sp]) for sp in ds})

Pin a revision for reproducible runs:

load_dataset("permitt/superglue-mne", "boolq", revision="v0.1.0-data")

Schema

Each example carries an integer row id idx and the task-specific input fields plus a label column. For cb:

{
    "idx": 0,
    "premise": "Poli je morala brzo da razmisli. ...",
    "hypothesis": "Poli nije bila iskusna okeanska jedriličarka.",
    "label": 0,  # 0=entailment, 1=contradiction, 2=neutral
}

Per-task field lists are documented in the BalkanBench source repo.

Methodology

• —Translation: original SuperGLUE English -> Montenegrin via translation + human verification by native speakers.
• —Frozen splits: train/validation/test row counts and IDs pinned at tag v0.1.0-data.
• —Test labels stay private: see the gated sibling repo above.

Citation

If you use this dataset, please cite the Serbian SuperGLUE paper (which introduces the BCMS adaptation methodology used here), the original SuperGLUE paper, and the BalkanBench release:

@inproceedings{perovic-mihajlov-2026-serbian,
    title = "{S}erbian {S}uper{GLUE}: Towards an Evaluation Benchmark for {S}outh {S}lavic Language Models",
    author = "Perovic, Mitar and Mihajlov, Teodora",
    editor = "Hettiarachchi, Hansi and Ranasinghe, Tharindu and Plum, Alistair and
              Rayson, Paul and Mitkov, Ruslan and Gaber, Mohamed and Premasiri, Damith and
              Tan, Fiona Anting and Uyangodage, Lasitha",
    booktitle = "Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.loreslm-1.30/",
    doi = "10.18653/v1/2026.loreslm-1.30",
    pages = "347--361",
    ISBN = "979-8-89176-377-7"
}

@inproceedings{wang2019superglue,
  title={SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems},
  author={Wang, Alex and Pruksachatkun, Yada and Nangia, Nikita and Singh, Amanpreet and
          Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.},
  booktitle={NeurIPS},
  year={2019}
}

@misc{balkanbench2026superglue_mne,
  title={BalkanBench SuperGLUE-MNE: Montenegrin SuperGLUE for the BCMS evaluation suite},
  author={Perović, Mitar and contributors},
  year={2026},
  howpublished={\url{https://huggingface.co/datasets/permitt/superglue-mne}},
  note={Part of BalkanBench v0.1, \url{https://balkanbench.com}}
}

License

CC-BY-4.0 (matches the original SuperGLUE source license).

Sponsor

Compute for the official v0.1 evaluation is sponsored by [Recrewty](https://recrewty.com).

Links

• —Live leaderboard: <https://balkanbench.com/leaderboard>
• —Project home: <https://balkanbench.com>
• —Background: Release of BalkanBench - the vision behind it (Medium)
• —Source code: <https://github.com/permitt/balkan-bench>
• —Sibling tracks: SuperGLUE-SR (Serbian, official) · SuperGLUE-HR (Croatian)
• —Private (gated) labels: <https://huggingface.co/datasets/permitt/superglue-mne-private>