helloadhavan/CC-FilteredCorpus

English Cleaned Common Crawl Markdown Dataset

An English-focused dataset created from Common Crawl, cleaned and converted to Markdown. The goal is to preserve web-document structure so that AI models can learn both natural language and Markdown formatting.

Features

• —English-focused
• —Cleaned and filtered web content
• —HTML converted to Markdown
• —Exact and near-duplicate filtering
• —GPT-2 perplexity filtering
• —Stored as compressed Parquet shards

Source

The current dataset is being created from:

Common Crawl — `CC-MAIN-2026-25`

Processing

The pipeline roughly follows:

Common Crawl
     ↓
HTML → Markdown
     ↓
Text cleaning
     ↓
English filtering
     ↓
Deduplication
     ↓
Quality filtering
     ↓
Parquet shards

Dataset Format

Each document contains:

• —text — cleaned Markdown
• —url — original URL
• —warc_date — Common Crawl date
• —record_id — WARC record ID
• —payload_digest — Common Crawl digest
• —edu_score — FineWeb-Edu score
• —perplexity — GPT-2 perplexity

Example:

{
  "text": "A strong first passage makes the page easier to scan and improves the sense that the story belongs to a broader publication. Page 541 keeps this section distinct through ordering, context and a stable archival route. The opening section frames the topic with a calmer editorial rhythm and gives the page a clear entry po...",
  "url": "http://aece.website/context-and-a-stable-archival-route-541/",
  "warc_date": "2026-06-05T23:15:05Z",
  "record_id": "urn:uuid:10c5fb6b-6359-4bc0-b503-217bb07ce3cb",
  "payload_digest": "sha1:6JK43QOH3CQ64XB3MUWUICDVWDXJUEAO",
  "edu_score": 1.767578125,
  "perplexity": 0.0
}

File format

Each common crawl timeline (Eg: 2025-26) is stored as a seperate folder like CC-MAIN-2025-26

Status

Processing in progress. The dataset is uploaded shard by shard while it is being processed. Because processing is still ongoing, the Hugging Face Data Viewer may not be available yet. However, individual Parquet shards can already be downloaded from the repository.

Intended Use

This dataset is mainly intended for research and experimentation involving:

Markdown-aware language models LLM pretraining Markdown generation Structured document generation Web-text research

<blockquote> This dataset is derived from publicly crawled web content. Individual documents may have different copyright or licensing conditions. Please check the applicable rights before using the data. The filtering pipeline is automated, so some unwanted or low-quality content may remain. </blockquote>

Feedback

Feedback on the cleaning, Markdown quality, filtering, and dataset format is welcome!