Symato/cc
What is Symato CC? To download all WARC data from Common Crawl then filter out Vietnamese in Markdown and Plaintext format. There is 1% of Vietnamse in CC, extract all of them out should be a lot (~10TB of plaintext). Main contributors https://huggingface.co/nampdn-ai https://huggingface.co/binhvq https://huggingface.co/th1nhng0 https://huggingface.co/iambestfeed Simple quality filters To make use of raw data from common crawl, you need to do… See the full description on the dataset page: https://huggingface.co/datasets/Symato/cc.
3414k
1---2license: mit3language:4- vi5size_categories:6- 1K<n<10K7---8 9# What is Symato CC?10To download all WARC data from Common Crawl then filter out Vietnamese in Markdown and Plaintext format.11There is 1% of Vietnamse in CC, extract all of them out should be a lot (~10TB of plaintext).12 13## Main contributors14- https://huggingface.co/nampdn-ai15- https://huggingface.co/binhvq16- https://huggingface.co/th1nhng017- https://huggingface.co/iambestfeed18 19# Simple quality filters20To make use of raw data from common crawl, you need to do filtering and deduping. 21Below is a simple quality filtering code for reference to write your own filters.22 23```sh24## Convert .parquet to .jsonl.gz25mkdir -p jsonl filtered26python3 parquet2jsonl.py27 28## Quality filter29# wget https://huggingface.co/datasets/Symato/goods_vs_c4_cc_classifiers/resolve/main/fasttext_good_vs_c4_001.bin 30python3 filters.py jsonl/2023-14_20230401125552-20230401155552.jsonl.gz logging31```32 33# Disclaimer34- We use content from Common Crawl as it is. Go to CC website to know more about data.35- We provide simple quality filters code to make it easier for you to use data but no warranty the data quality meet everyone expectations. Modifiy ours or write your own filters in-case you need more advanced / better ones.36 37Contact **dung at symato dot xyz** if you have other questions.38