EB-Sky/python-domain-10gb
EB-Sky Python Domain (10GB) ~10GB of raw Python domain knowledge for continued pretraining: ~8GB of source code plus Stack Overflow Python Q&A. (2.49GB as stored — zstd-compressed parquet shards.) Sources Code: codeparrot/codeparrot-clean Q&A: koutch/stackoverflow_python Processing Size filter (200–200,000 chars per document) Auto-generated file removal (header heuristics) Hardcoded-secret regex scan, email redaction Exact deduplication (SHA-1)… See the full description on the dataset page: https://huggingface.co/datasets/EB-Sky/python-domain-10gb.
053
1---2license: other3task_categories:4- text-generation5language:6- code7- en8size_categories:9- 1M<n<10M10pretty_name: EB-Sky Python Domain 10GB11---12 13# EB-Sky Python Domain (10GB)14 15~10GB of raw Python domain knowledge for continued pretraining:16~8GB of source code plus Stack Overflow Python Q&A.17(2.49GB as stored — zstd-compressed parquet shards.)18 19## Sources20- **Code:** [codeparrot/codeparrot-clean](https://huggingface.co/datasets/codeparrot/codeparrot-clean)21- **Q&A:** [koutch/stackoverflow_python](https://huggingface.co/datasets/koutch/stackoverflow_python)22 23## Processing24- Size filter (200–200,000 chars per document)25- Auto-generated file removal (header heuristics)26- Hardcoded-secret regex scan, email redaction27- Exact deduplication (SHA-1)28- Q&A: HTML stripped, minimum score 1 for question and answer29 30## Format31| Column | Description |32|---|---|33| `text` | the document content |34| `source` | `github-code` or `stackoverflow-qa` |35 36## Load it37```python38from datasets import load_dataset39ds = load_dataset("EB-Sky/python-domain-10gb", split="train")