CoolFace
Datasetpublic

EB-Sky/python-domain-10gb

EB-Sky Python Domain (10GB) ~10GB of raw Python domain knowledge for continued pretraining: ~8GB of source code plus Stack Overflow Python Q&A. (2.49GB as stored — zstd-compressed parquet shards.) Sources Code: codeparrot/codeparrot-clean Q&A: koutch/stackoverflow_python Processing Size filter (200–200,000 chars per document) Auto-generated file removal (header heuristics) Hardcoded-secret regex scan, email redaction Exact deduplication (SHA-1)… See the full description on the dataset page: https://huggingface.co/datasets/EB-Sky/python-domain-10gb.

sourceHugging Faceotherupdated 22h agoView on Hugging Face
0likes53downloads
Dataset Card

EB-Sky Python Domain (10GB)

~10GB of raw Python domain knowledge for continued pretraining: ~8GB of source code plus Stack Overflow Python Q&A. (2.49GB as stored — zstd-compressed parquet shards.)

Sources

Processing

  • —Size filter (200–200,000 chars per document)
  • —Auto-generated file removal (header heuristics)
  • —Hardcoded-secret regex scan, email redaction
  • —Exact deduplication (SHA-1)
  • —Q&A: HTML stripped, minimum score 1 for question and answer

Format

ColumnDescription
textthe document content
sourcegithub-code or stackoverflow-qa

Load it

python
from datasets import load_dataset
ds = load_dataset("EB-Sky/python-domain-10gb", split="train")