agentlans/common-crawl-sample

Common Crawl sample

A small unofficial random subset of the famous Common Crawl dataset.

• —60 random segment WET files were downloaded from Common Crawl on 2024-05-12.
• —Lines between 500 and 5000 characters long (inclusive) were kept.
• —Only unique texts were kept.
• —No other filtering.

Languages

• —Each text was assigned to one of the language codes using the GCLD3 Python package.
• —The Chinese texts were classified as either simplified, traditional, or Cantonese using the fastlangid package.
• —For each language, 10% of the rows were randomly selected as the test set.
• —The test set of the "all" languages split is the union of the test sets of all the languages in the dataset.

⚠️ Warning! 🚫

• —This dataset is raw and unfiltered from the Internet.
• —So it contains objectionable content, false information, and possibly personally identifiable information.
• —But it's mostly spam and repetitive junk. Just spam, spam, spam. Spam everywhere. 💩 Please filter it according to your needs.

Limitations

• —Some languages are greatly overrepresented.
• —Samples of webpages may not represent real language use.
• —Major problem is spam. Throws off the language detectors.
• —Many false positives like Taiwanese traditional Chinese being classified as Cantonese.
• —The testing split isn't truly independent of the training split
• —for example, different paragraphs from the same webpage can end up in both training and testing splits