nawabhussain/Kashmiri-Language-Corpus

Kashmiri Textual Data Corpus

Introduction

This repository contains a combined dataset of Kashmiri textual data collected from various sources. The data has been sourced from different locations and may contain non-Kashmiri text (e.g., Urdu, Persian). The goal of this corpus is to provide a wide variety of Kashmiri text data for research and language processing tasks.

Sources of Data

1. mzmmoazam/kashmiri_dataset (HTML Data)

Source: GitHub - mzmmoazam/kashmiri_dataset

This dataset contains Kashmiri text scraped from various HTML pages. Some sentences may include non-Kashmiri text.

Data Sample: | sentence | source | |----------|--------| | یہ چھہ دوٚیم حصہ گۅڈنیک حصہ وُچھِو * تۯیم حصہ وُچھِو کتاب ٹھیک پٲٹھۍ پرنہ باپتھ چھہ گلمرگ نستعلیق فانٹ انسٹال کرن ضروری۔ | https://github.com/mzmmoazam/kashmiridataset | | سرمایا تہٕ سام | https://github.com/mzmmoazam/kashmiridataset |

2. SMUQamar

Source: Hugging Face - SMUQamar/Kashmiri-English-Parallel-Corpus

This dataset contains Kashmiri-English parallel sentences.

Data Sample: | sentence | source | |----------|--------| | کیٛاہ ژٕ ہؠکہٕ مےٚ کینٛہہ ہیلدی فوڈ ڈِنَر خٲطرٕ سجسٹہٕ کٔرِتھ | https://huggingface.co/datasets/SMUQamar/Kashmiri-English-Parallel-Corpus | | مہربٲنی کٔرِتھ کر سپیکرچ آواز کم | https://huggingface.co/datasets/SMUQamar/Kashmiri-English-Parallel-Corpus |

3. Wikipedia

Source: Kashmiri Wikipedia

Data scraped from Kashmiri Wikipedia articles.

Data Sample: | sentence | source | |----------|--------| | 1- (نَمبَر) چھُ اَکھ نَمبَر یُس -2 کھوتہٕ بۆڈ چھُ تہٕ 0 کھوتہٕ لوکُٹ چھُ۔ | https://ks.wikipedia.org | | 0 (نَمبَر) یا صِفَر چھُ اَکھ نَمبَر یُس -1 کھوتہٕ بۆڈ چھُ تہٕ 1 کھوتہٕ لوکُٹ چھُ. | https://ks.wikipedia.org |

4. NLLB

Source: Hugging Face - allenai/nllb

This dataset contains Kashmiri sentences from the NLLB project.

Data Sample: | sentence | source | |----------|--------| | اسہِ تہِ ؤژھ آکاشہِ برکت تٲے تٲے | https://huggingface.co/datasets/allenai/nllb | | رٲتۍ راتس پتھ عذابن میٲنۍ تن وٮتراں کُنی | https://huggingface.co/datasets/allenai/nllb |

5. OpenSlr

Source: OpenSlr - Dataset 122

This dataset contains Kashmiri sentences collected from OpenSlr.

Data Sample: | sentence | source | |----------|--------| | مِہ ہز چھ ناو آدِل بشیر | https://openslr.org/122/ | | بٔہ ہز چُھس بسان کۅلگامِہ ڈِسٹِک | https://openslr.org/122/ |

Data Preprocessing

The data has been processed by performing the following steps:

1. 1.Merging Data: All datasets were combined into a single dataset.
2. 2.Cleaning: Duplicates were removed, and sentences with very few words were filtered out.
3. 3.Word Count: Word counts for each sentence were calculated and visualized.
4. 4.Filtered Data: Sentences with fewer than 3 words were excluded from the final dataset.

Cleaned Data Statistics:

• —Total Sentences: 46,748
• —Average Word Count per Sentence: 16.1
• —Max Word Count per Sentence: 6767
• —Min Word Count per Sentence: 3

Citation

If you use this dataset in your research, please cite it as:

@misc{
KashmiriCorpus2024,
author = {Nawab},
title = {Kashmiri Textual Data Corpus},
year = {2024},
url = {https://huggingface.co/datasets/kashmiri_corpus} 
}