CoolFace
Datasetpublic

G-reen/cc-2020-raw

cc-2020-raw English web documents extracted from the Common Crawl CC-MAIN-2020-50 snapshot, intended as a pre-2022 human-authored text corpus (i.e. crawled before generative-model output became widespread on the web). Pipeline Stream — Common Crawl WET records, prefiltered on length, replacement-character ratio, and printable/alphabetic character ratios. Language filter — langdetect at p >= 0.95 on three sampled spans of each document; all spans must be English.… See the full description on the dataset page: https://huggingface.co/datasets/G-reen/cc-2020-raw.

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face
0likes7downloads
Dataset Card

cc-2020-raw

English web documents extracted from the Common Crawl CC-MAIN-2020-50 snapshot, intended as a pre-2022 human-authored text corpus (i.e. crawled before generative-model output became widespread on the web).

Pipeline

  1. 1.Stream — Common Crawl WET records, prefiltered on length, replacement-character ratio, and printable/alphabetic character ratios.
  2. 2.Language filter — langdetect at p >= 0.95 on three sampled spans of each document; all spans must be English.
  3. 3.Extraction — the matching WARC record is fetched and the HTML re-extracted with Trafilatura 2.0.0 (favor_precision), keeping documents of at least 200 words.
  4. 4.Dedup — exact normalized-text hash, canonical URL, and 64-bit SimHash near-duplicate removal at Hamming distance <= 3.
  5. 5.Cross-snapshot dedup — this snapshot is the primary; documents shared with the other snapshot are kept here.

Fields

fielddescription
cc_snapshotCommon Crawl snapshot id
urlcanonical source URL
timestampcrawl timestamp
textraw WET plain text
word_countwhitespace token count of text
trafilatura_texttext re-extracted from the archived HTML
trafilatura_word_countwhitespace token count of trafilatura_text
raw_langdetect_prob / trafilatura_langdetect_probEnglish confidence
wet_path / html_warc_path / record_id / html_record_idprovenance back to Common Crawl
text_sha256 / trafilatura_text_sha256content hashes

trafilatura_text is generally the field you want; text is kept for reference.

Stats

  • —Documents: 5,960,628
  • —Uncompressed: ~91 GB across 23 shards

Licensing

Derived from Common Crawl, which is distributed under its Terms of Use. Individual documents remain subject to the rights of their original publishers.