datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
c4-en-html-with-metadatac4-en-html-with-metadata-ppl-cleanFile list:
"c4-en-html_cc-main-2019-18_pq00-000.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-001.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-002.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-003.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-004.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-005.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-006.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-007.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-008.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-009.jsonl.gz"… See the full description on the dataset page: https://huggingface.co/datasets/masoudjs/c4-en-html-with-metadata-ppl-clean.c4-en-html-with-training_metadata_allchunked-enwiki-ns0-20250301-enterprise-html20250301.jsonl contains preprocessed text chunks from the wikipedia 20250301 html dump.
dead_letter_queue.jsonl contains articles that couldn't be processed due to formatitng or missing tags.
en_redirection_map.pkl can be deserialize to avoid building redirection_map everytime preprocessing_wikipedia_html_dump runs.
chunked-enwiki-ns0-20250301-enterprise-html20250301.jsonl contains preprocessed text chunks from the wikipedia 20250301 html dump.
dead_letter_queue.jsonl contains articles that couldn't be processed due to formatitng or missing tags.
en_redirection_map.pkl can be deserialize to avoid building redirection_map everytime preprocessing_wikipedia_html_dump runs.