permitt/superglue-hr

BalkanBench SuperGLUE - Croatian

Part of [BalkanBench](https://balkanbench.com) - the open, reproducible benchmark for language models across Serbian, Croatian, Montenegrin, and Bosnian (BCMS). Live leaderboard at <https://balkanbench.com/leaderboard>. Background: Release of BalkanBench - the vision behind it (Medium, 2026-04-27).

This is the Croatian SuperGLUE released preview track of BalkanBench v0.1. Croatian publishes alongside Serbian (the official frozen track) and Montenegrin as a 5-task preview: same scoring contract, same private test labels, no WSC adaptation yet.

What's inside

5 task configs, full train + validation + public test splits (train and validation labelled here; the test split lives only in the gated sibling repo, see below):

Total: 65,049 items. WSC and the AX-b/AX-g diagnostics are Serbian-only in v0.1 - they may be added in a later release once a Croatian adaptation is available.

COPA source attribution: the copa config is adapted from classla/copa_hr (COPA-HR, CLARIN.SI, CC-BY-SA-4.0), the Croatian COPA released by the CLASSLA team as part of the CLARIN.SI South Slavic COPA family alongside COPA-SR and COPA-MK. See the Citation section at the end of this card for the full reference.

Test split is gated

The test split for every config lives only in the gated sibling repo (both inputs and labels): `permitt/superglue-hr-private`, and are accessed only by the official scoring pipeline. To get a leaderboard score you generate predictions on the public test inputs with balkanbench predict, then submit through the submission flow.

Loading

from datasets import load_dataset

boolq = load_dataset("permitt/superglue-hr", "boolq")
print(boolq["train"][0])

# All ranked tasks
for cfg in ["boolq", "cb", "copa", "rte", "multirc"]:
    ds = load_dataset("permitt/superglue-hr", cfg)
    print(cfg, {sp: len(ds[sp]) for sp in ds})

Pin a revision for reproducible runs:

load_dataset("permitt/superglue-hr", "boolq", revision="v0.1.0-data")

Schema

Each example carries an integer row id idx and the task-specific input fields plus a label column. For cb:

{
    "idx": 0,
    "premise": "Polly je morala brzo razmišljati. ...",
    "hypothesis": "Polly nije bila iskusna oceanska jedriličarka",
    "label": 0,  # 0=entailment, 1=contradiction, 2=neutral
}

Per-task field lists are documented in the BalkanBench source repo.

Methodology

• —Translation: original SuperGLUE English -> Croatian via translation + human verification by native speakers.
• —Frozen splits: train/validation/test row counts and IDs pinned at tag v0.1.0-data.
• —Test labels stay private: see the gated sibling repo above.

License

CC-BY-4.0 (matches the original SuperGLUE source license). Note: the copa config is adapted from a CC-BY-SA-4.0 source (classla/copa_hr, see Citation below); redistribution of that config should retain the ShareAlike attribution.

Sponsor

Compute for the official v0.1 evaluation is sponsored by [Recrewty](https://recrewty.com).

Links

• —Live leaderboard: <https://balkanbench.com/leaderboard>
• —Project home: <https://balkanbench.com>
• —Background: Release of BalkanBench - the vision behind it (Medium)
• —Source code: <https://github.com/permitt/balkan-bench>
• —Sibling tracks: SuperGLUE-SR (Serbian, official) · SuperGLUE-MNE (Montenegrin)
• —Private (gated) labels: <https://huggingface.co/datasets/permitt/superglue-hr-private>

Citation

If you use this dataset, please cite the Serbian SuperGLUE paper (which introduces the BCMS adaptation methodology used here), the original SuperGLUE paper, and the BalkanBench release. If you use the copa config specifically, please also cite the classla COPA-HR source dataset it is adapted from:

@inproceedings{perovic-mihajlov-2026-serbian,
    title = "{S}erbian {S}uper{GLUE}: Towards an Evaluation Benchmark for {S}outh {S}lavic Language Models",
    author = "Perovic, Mitar and Mihajlov, Teodora",
    editor = "Hettiarachchi, Hansi and Ranasinghe, Tharindu and Plum, Alistair and
              Rayson, Paul and Mitkov, Ruslan and Gaber, Mohamed and Premasiri, Damith and
              Tan, Fiona Anting and Uyangodage, Lasitha",
    booktitle = "Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.loreslm-1.30/",
    doi = "10.18653/v1/2026.loreslm-1.30",
    pages = "347--361",
    ISBN = "979-8-89176-377-7"
}

@inproceedings{wang2019superglue,
  title={SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems},
  author={Wang, Alex and Pruksachatkun, Yada and Nangia, Nikita and Singh, Amanpreet and
          Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.},
  booktitle={NeurIPS},
  year={2019}
}

@misc{balkanbench2026superglue_hr,
  title={BalkanBench SuperGLUE-HR: Croatian SuperGLUE for the BCMS evaluation suite},
  author={Perović, Mitar and contributors},
  year={2026},
  howpublished={\url{https://huggingface.co/datasets/permitt/superglue-hr}},
  note={Part of BalkanBench v0.1, \url{https://balkanbench.com}}
}

@article{DBLP:journals/corr/abs-2104-09243,
 author    = {Nikola Ljube{\v s}i{\'c} and
              Davor Lauc},
 title     = {BERTi{\'c} - The Transformer Language Model for Bosnian, Croatian,
               Montenegrin and Serbian},
 journal   = {CoRR},
 volume    = {abs/2104.09243},
 year      = {2021},
 url       = {https://arxiv.org/abs/2104.09243},
 archivePrefix = {arXiv},
 note      = {Cited by classla/copa_hr (COPA-HR) as its requested reference}
}