anandjh8/common-crawl-english-filtered

🧠 FineWeb-English-Filtered

📘 Dataset Summary

FineWeb-English-Filtered is a large-scale, cleaned, English-only text dataset derived from Common Crawl’s WET archives. It contains 940 million documents of publicly available web text, converted into Apache Parquet format with a consistent schema for fast and efficient data loading.

The dataset was generated using a custom AWS Glue pipeline that processed, filtered, and merged .wet files across multiple terabytes of Common Crawl data.

This dataset is ideal for training large language models, retrieval research, and web-scale NLP tasks.

📂 Data Structure

Each record corresponds to one web document extracted from a WARC/WET file.

⚙️ Data Processing Pipeline

1. 1.Extraction – Read .wet files from AWS S3 (s3://warc-unc-*).
2. 2.Filtering – Retain only records where WARC-Identified-Content-Language == 'eng'.
3. 3.Schema Enforcement – Normalize metadata fields across all records.
4. 4.Serialization – Store each record as a Parquet row using PyArrow.
5. 5.Parallel Processing – Processed with AWS Glue (Spark) and Boto3.
6. 6.Merging – Folder-wise merges (~290 GB per shard) into unified Parquet outputs.
7. 7.Upload – Finalized dataset uploaded to Hugging Face Hub.

🧩 Technical Specifications

💡 Example Usage

from datasets import load_dataset

ds = load_dataset("anandjha8/fineweb-english-filtered", split="train")

print(ds)
print(ds.features)
print(ds[0])