sobir-hf/tajik-text-segmentation
This dataset contains texts in Tajik language with sentence annotations. It can be used to train and evaluate sentence-wise text segmentation algorithms. The dataset contains more than 100 short and long texts and more than 3000 annotated sentences. The texts were carefully selected from different catergories such as news, articles, novels, classical texts, poetry, and religious texts. It deliberately contains more of "hard" passages where splitting them by period "." characters would result… See the full description on the dataset page: https://huggingface.co/datasets/sobir-hf/tajik-text-segmentation.
strip sentence ends when parsing
corrected more annotation mistakes
fixed some annotation mistakes
bugfix where # was interfering in parsing
updated format and data
Update README.nd
converted the files to jsonl
Update tajik-text-segmentation.py
Update tajik-text-segmentation.py
Create README.md
Update tajik-text-segmentation.py
Update tajik-text-segmentation.py
Upload tajik-text-segmentation.py
Uploaded data files and parser script
initial commit