nickoo004/kaa-parallel-corpus

Kaa Karakalpak-English Parallel Corpus (FineTranslations)

📌 Overview

This repository contains a high-quality, curated parallel corpus for the Karakalpak (kaa) language, paired with English (en). Karakalpak is a low-resource Turkic language spoken primarily in the Republic of Karakalpakstan.

This dataset is a specialized subset extracted from the massive [HuggingFaceFW/finetranslations](https://huggingface.co/datasets/HuggingFaceFW/finetranslations) project. The goal of this repo is to provide a dedicated and easy-to-access resource for researchers and developers focusing specifically on Karakalpak NLP.

🗂️ Dataset Structure

The corpus is organized into two distinct configurations based on the writing systems used in Karakalpakstan:

Column Descriptions:

• —`id`: Unique identifier for the pair.
• —`translated_text`: The high-quality English translation.
• —`og_full_text`: The original Karakalpak sentence (Target).
• —`og_language`: The specific script tag (kaa_Cyrl or kaa_Latn).
• —`og_quality_score`: Quality metric from the base dataset.
• —`edu_score`: Educational value score (higher means better content quality).
• —`url`: Source web address.

🚀 Usage

You can load the dataset using the Hugging Face datasets library:

1. Load Cyrillic Data

from datasets import load_dataset

dataset = load_dataset("nickoo004/kaa-parallel-corpus", "kaa_Cyrl", split="train")
print(dataset[0])

2. Load Latin Data

dataset_latn = load_dataset("nickoo004/kaa-parallel-corpus", "kaa_Latn", split="train")

3. Quick Format for Training (MT)

If you need a simple en-kaa format for training models like NLLB or T5:

def format_data(example):
    return {
        "english": example["translated_text"],
        "karakalpak": example["og_full_text"]
    }

clean_ds = dataset.map(format_data, remove_columns=dataset.column_names)

🛠️ Data Origin & Quality

The data was collected and processed by the Hugging Face FineData Team as part of the FineTranslations effort. The pipeline included:

1. 1.Crawling: Extracting Karakalpak text from diverse web sources (Wikipedia, government sites, news).
2. 2.Filtering: Removing low-quality or non-Karakalpak content using advanced language identification.
3. 3.Translation: Generating high-saliency English translations using state-of-the-art translation models.
4. 4.Scoring: Annotating rows with educational and quality scores to allow for better training data selection.

📜 License & Citation

This dataset is released under the Apache-2.0 License.

If you use this dataset, please credit the original source:

@software{finetranslations2024,
  author = {Hugging Face FineData Team},
  title = {FineTranslations: A Large-Scale High-Quality Parallel Corpus},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face Repository},
  howpublished = {\url{https://huggingface.co/datasets/HuggingFaceFW/finetranslations}}
}

Maintained by: nickoo004 Contact: nursultankoshekbaev477@gmail.com