malayalam
Malayalam_CultureX_IndicCorp_SMCMalayalam Pretraining/Tokenization dataset.
Preprocessed and combined data from the following links,
* ai4bharat
* CulturaX
* Swathanthra Malayalam Computing
Commands used for preprocessing.
To remove all non Malayalam characters.
sed -i 's/[^ം-ൿ.,;:@$%+&?!() ]//g' test.txt
To merge all the text files in a particular Directory(Sub-Directory)
find SMC -type f -name '*.txt' -exec cat {} ; >> combined_SMC.txt
To remove all lines with characters less than 5.
grep -P… See the full description on the dataset page: https://huggingface.co/datasets/VishnuPJ/Malayalam_CultureX_IndicCorp_SMC.asr_malayalamOCR-bench-Malayalammalayalam_2020_wiki��This dataset is from the common-crawl-malayalam repo: https://github.com/qburst/common-crawl-malayalam
indic-Malayalam-PDSPRING_INX_Malayalam_R1