datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
sentiment-dksfThe Sentiment DKSF (Digikala/Snappfood comments) is a dataset for sentiment analysis.
persian-license-plate-v1
Dataset is downloaded from here which was provided at Amirkabir University of Technology.
The dataset is labeled by the authors.
Experimental results show that the fine-tuned model works well in Persian License Plate.
Usage
You can download the dataset easily using HF datasets package in Python:
!pip install datasets
from datasets import load_dataset
dataset = load_dataset("hezarai/persian-license-plate-v1", split="train") # Other splits: validation, test
print(dataset[0])
lscp-pos-500kThis is a 500 thousand sample version of the original LSCP dataset that only contains the text and part-of-speech tags and is used for sequence labeling.
Citation
@InProceedings{abdikhojasteh:2020:LREC,
author = {Abdi Khojasteh, Hadi and Ansari, Ebrahim and Bohlouli, Mahdi},
title = {LSCP: Enhanced Large Scale Colloquial Persian Language Understanding},
booktitle = {Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC 2020)}… See the full description on the dataset page: https://huggingface.co/datasets/hezarai/lscp-pos-500k.flickr30k-faThe Flickr30K dataset filtered and translated to Persian.
This dataset was originally made by Sajjad Ayoubi and uploaded to Kaggle at https://www.kaggle.com/datasets/sajjadayobi360/flickrfa.
This repo contains the exact dataset split to train/test using a custom sampling criteria and can be directly loaded using HuggingFace datasets or right from Hezar.
Usage
Hugging Face Datasets
pip install datasets
from datasets import load_dataset
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/hezarai/flickr30k-fa.xlsum-faThe Persian portion of the XLSum dataset.
Citation
@inproceedings{hasan-etal-2021-xl,
title = "{XL}-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages",
author = "Hasan, Tahmid and
Bhattacharjee, Abhik and
Islam, Md. Saiful and
Mubasshir, Kazi and
Li, Yuan-Fang and
Kang, Yong-Bin and
Rahman, M. Sohel and
Shahriyar, Rifat",
booktitle = "Findings of the Association for Computational Linguistics:… See the full description on the dataset page: https://huggingface.co/datasets/hezarai/xlsum-fa.common-voice-13-faThe Persian portion of the original CommonVoice 13 dataset at https://huggingface.co/datasets/mozilla-foundation/common_voice_13_0
Load
# Using HF Datasets
from datasets import load_dataset
dataset = load_dataset("hezarai/common-voice-13-fa", split="train")
# Using Hezar
from hezar.data import Dataset
dataset = Dataset.load("hezarai/common-voice-13-fa", split="train")
parsynth-ocr-200kParsynthOCR is a synthetic dataset for Persian OCR. This version is a preview of the original 4 million samples dataset (ParsynthOCR-4M).
Usage
🤗 Datasets
from datasets import load_dataset
dataset = load_dataset("hezarai/parsynth-ocr-200k")
Hezar
pip install hezar
from hezar.data import Dataset
dataset = Dataset.load("hezarai/parsynth-ocr-200k", split="train")
parsynth-ocr-4m
Synthetic Persian OCR Dataset (4M)
This dataset contains 4,000,000 synthetic Persian text line images generated using a customized Persian fork of TextRecognitionDataGenerator (trdg). It is designed specifically for training high-performance optical character recognition (OCR) and text recognition models for Persian text.
This dataset served as the foundational pre-training dataset for the official Hezar AI Persian OCR models, including hezarai/crnn-base-fa-v2 and… See the full description on the dataset page: https://huggingface.co/datasets/hezarai/parsynth-ocr-4m.coco-flickr-faTranslated and filtered COCO 2017 and Flickr30K created by Sajjad Ayoubi & Navid Kanaani . (Original link)
The original dataset was split by a 80/20 train/test ratio with no further processing.
parstwinerParsTwiNER dataset created by Aghajani et al. Paper
As a result of unstructured sentences and some misspellings and errors, finding named entities in a noisy environment such as social media takes much more effort.
ParsTwiNER contains about 250k tokens, based on standard instructions like MUC-6 or CoNLL 2003, gathered from Persian Twitter. Using Cohen’s Kappa coefficient, the consistency of annotators is 0.95, a high score.
In this study, we demonstrate that some state-of-the-art models… See the full description on the dataset page: https://huggingface.co/datasets/hezarai/parstwiner.arman-nerhezarhezarAQHezarfen