Imsidag-community/nllb_en_kab

NLLB English - Kabyle Dataset

This dataset contains parallel sentences in English and Kabyle, cleaned and filtered using the GlotLid model. The dataset is derived from the OPUS-NLLB corpus and has been processed to ensure high-quality sentence pairs.

Dataset Structure

• —nllb_en_kab.parquet: A Parquet file containing the cleaned English-Kabyle sentence pairs.

Dataset Statistics

• —Total Sentence Pairs: 2,484,297
• —English Sentences: 2,484,297
• —Kabyle Sentences: 2,484,297

Data Preprocessing

1. 1.Source: The dataset is derived from the OPUS-NLLB corpus.
2. 2.Language Detection: The GlotLid v3 model was used to filter out non-Kabyle sentences. Only sentences with a confidence score of 0.95 or higher were kept.
3. 3.Normalization: A custom fixer/normalizer was used to correct non-standard Kabyle Latin characters. Such as the following replacements were made:
4. 4.ţţ → tt
5. 5.țț → tt
6. 6.ε → ɛ
7. 7.ϵ → ɛ
8. 8.γ → ɣ
9. 9.Γ → Ɣ
10. 10.Σ → Ɛ
11. 11.Ԑ → Ɛ
12. 12.ğ → ǧ
13. 13.ş → ṣ

Usage

You can load this dataset using the datasets library:

from datasets import load_dataset

dataset = load_dataset("Imsidag-community/nllb_en_kab", split="train")
print(dataset)

License

This dataset is licensed under the CC-BY-NC 4.0 License.

Citation

If you use this dataset in your research, please cite it as follows:

@dataset{
    nllb_en_kab,
    title = {NLLB English-Kabyle Dataset},
    authors = {Athmane MOKRAOUI},
    year = {2025},
    url = {https://huggingface.co/datasets/Imsidag-community/nllb_en_kab}
}