turkish-nlp-suite/TrGLUE

TrGLUE - The First Non-Translate Natural Language Understanding Benchmark for Turkish

<img src="https://raw.githubusercontent.com/turkish-nlp-suite/.github/main/profile/trgluelogo.png" width="30%" height="30%">

Dataset Card for TrGLUE

TrGLUE is a natural language understanding benchmarking dataset including several single sentence and sentence pair classification tasks. The inspiration is clearly the original GLUE benchmark.

Tasks

Single Sentence Tasks

TrCOLA The original Corpus of Linguistic Acceptability consists of sentences compiled from English literature textbooks. The task is to determine if the sentences are grammatically correct and acceptable sentences. Our corpus is also compiled from Turkish linguistic textbooks and include morphological, syntactic and semantic violations. This dataset also has a standalone repo on HuggingFace.

TrSST-2 The Stanford Sentiment Treebank is a sentiment analysis dataset includes sentences from movie reviews, annotated by human annotators. The task is to predict the sentiment of a given sentence. This dataset is compiled from movie review websites BeyazPerde.com and Sinefil.com, both reviews and sentiment ratings are compiled from those websites. Here we offer a binary classification task to be compatible with the original GLUE task, however we offer a 10-way classification challenge in this dataset's standalone HuggingFace repo.

Sentence Pair Tasks

TrMRPC The Microsoft Research Paraphrase Corpus is a dataset of sentence pairs automatically extracted from online news sources, with human annotations. The task is to determine whether the sentences are semantically equivalent. Our dataset is compiled from Turkish news corpus Havadis and constructed in a similar way to original dataset.

TrSTS-B The Semantic Textual Similarity Benchmark is a semantic similarity dataset. This dataset contains sentence pairs compiled from news headlines, video and image captions. Each pair is annotated with a similarity score from 1 to 5. Our dataset is a revised version of the translation of this dataset.

TrQQP The Quora Question Pairs2 dataset is a collection of question pairs from Quora website. The task is to determine whether a pair of questions are semantically equivalent. Our dataset is compiled from several question-answer websites such as KizlarSoruyor.com, Sorucevap.com and similar.

TrMNLI The Multi-Genre Natural Language Inference Corpus is a dataset for the textual entailment task. The dataset is crowsourced. Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis, contradicts the hypothesis (contradiction), or neither (neutral). The premise sentences are compiled from different sources, including transcribed speech, fiction writings, and more. Our dataset is compiled in a similar fashion, the data source is OzenliDerlem, including fairy tales, viral media, social blogging content etc.

TrQNLI The Stanford Question Answering Dataset (SQuAD) is a well-known question-answering dataset consisting of context-question pairs, where the context text (drawn from Wikipedia) contains the answer to the corresponding question (written by an annotator). QNLI is a binary classification dataset version of SQuAD, where the task is to classify the context text includes the answer to the question text. Our dataset is compiled in a similar fashion. We built a SQuAD like dataset (to be published later in its own repo), then converted context-question-answer pairs to QNLI format.

TrRTE The Recognizing Textual Entailment dataset is compiled from a series of annual textual entailment challenges namely RTE1, RTE3 and RTE5. The task is again textual entailment. Our dataset is similarly constructed, first we compile data from newspaper and academical writing text, then constructed the hypothis text.

Lack of TrWNLI The Winograd Schema Challenge, introduced by Levesque et al. in 2011, is a type of reading comprehension task. In this challenge, a system is tasked with reading a sentence containing a pronoun and determining the correct referent for that pronoun from a set of choices. These examples are deliberately designed to outsmart basic statistical methods by relying on contextual cues provided by specific words or phrases within the sentence. To transform this challenge into a sentence pair classification task, the creators of the benchmark generate pairs of sentences by replacing the ambiguous pronoun with each potential referent. The objective is to predict whether the sentence remains logically consistent when the pronoun is substituted with one of the choices. We do not include this task in our benchmark as this task does not make sense for Turkish at all due to pro-drop nature of Turkish. For the details and examples please visit the research paper.

Dataset Statistics

The sizes of each dataset are as below:

Note that STS-B has a test split in the repo, this split is placed only for convenience purposes (so that tets split is not empty in TrGLUE evaluation script.) For the evaluation, please use validation split. For more information about dataset statistics, please visit the [research paper]().

Dataset Curation

Almost all the datasets are constructed from scratch, except TrSTS-B. The reason behind this choice is the dataset sentences come from video and image headlines, describing actions and situations; hence mostly cross-language compatible. Yet after the translation we made a supervision round. Rest of the datasets are constructed as follows:

• —The data is compiled from Turkish corpora.
• —We assign coarse labels to sentence pairs with some semantic similarity methods.
• —Then we ask our LLM of choice Arctic Snowflake to generate labels.
• —We compare coarse labels to LLM labels and feed non-matches to human annotators.

There are some exceptions still:

• —TrSST-2 is scraped from Turkish movie review websites, Sinefil and Beyazperde.
• —TrCOLA is compiled from openly available linguistic books, then generated violations by the LLM Snowflake Arctic and then curated by the data company Co-one. For more information please refer to the TrCOLA's standalone repo and the [research paper]().

For more information about the construction process and annotation guidelines please visit the research paper.

Benchmarking

We provide benchmarking script at TrGLUE Github repo. The script is the same with HF's original benchmarking script, for TrSST-2 we report F1-score in addition to accuracy due to class imbalance as well. We offered hyperparameters for reproducing our results as well.

We benchmarked BERTurk on all of our datasets:

The comparison with BERT on original GLUE is given in the research paper. Also we benchmarked a handful of popular LLMs on challenging sets such as TrCOLA and Turkish Hate Map, for the evaluations please visit the research paper as well.

Publication

is under review, preprint available at: Arxiv preprint

Blog posts

Available at our Github blog

Medium blog post is available at GDE Blog as well.

Citation

Preprint citation:

@misc{altinok2025introducingtrgluesentiturcacomprehensive,
      title={Introducing TrGLUE and SentiTurca: A Comprehensive Benchmark for Turkish General Language Understanding and Sentiment Analysis}, 
      author={Duygu Altinok},
      year={2025},
      eprint={2512.22100},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.22100}, 
}