stefan-it/turblimp-evaluations
TurBLiMP Evaluations This dataset hosts the TurBLiMP evaluation results on my Turkish Model Zoo. More about the TurBLiMP benchmark: TurBLiMP is the first Turkish benchmark of linguistic minimal pairs, designed to evaluate the linguistic abilities of monolingual and multilingual language models (LMs). This benchmark covers 16 core grammatical phenomena in Turkish, with 1,000 minimal pairs per phenomenon. Additionally, it incorporates experimental paradigms that examine model… See the full description on the dataset page: https://huggingface.co/datasets/stefan-it/turblimp-evaluations.
TurBLiMP Evaluations
This dataset hosts the TurBLiMP evaluation results on my Turkish Model Zoo.
More about the TurBLiMP benchmark:
TurBLiMP is the first Turkish benchmark of linguistic minimal pairs, designed to evaluate the linguistic abilities of monolingual and multilingual language models (LMs). This benchmark covers 16 core grammatical phenomena in Turkish, with 1,000 minimal pairs per phenomenon. Additionally, it incorporates experimental paradigms that examine model performance across different subordination strategies and word order variations.
I've modified the original evaluation script and extended it for the Turkish Model Zoo. My evaluation script can be found here.
Results
After running the evaluation script, all results can be parsed with this notebook to print-out a nice overview table:
Summary
The TurBLiMP paper used the `dbmdz/bert-base-turkish-128k-uncased` for evaluation, yielding a strong performance.
My evaluation here showed, that the `dbmdz/bert-base-turkish-128k-cased` even performs better on the TurBLiMP benchmark.