Datasetpublic

timaeus/dsir-pile-13m-filtered-no-github-or-dm_mathematics

DSIR Pile 13M - Filtered Version This is a filtered version of timaeus/dsir-pile-13m. Filtering Applied: Excluded: All rows where metadata.pile_set_name contains 'Github' or 'DM_mathematics' Kept: All other rows from the original dataset Dataset Size Original: ~13M examples Filtered: 12,782,200 examples (99.9% of original) Uploaded in: 64 batch files Usage from datasets import load_dataset dataset =… See the full description on the dataset page: https://huggingface.co/datasets/timaeus/dsir-pile-13m-filtered-no-github-or-dm_mathematics.

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes399downloads

Dataset Card

DSIR Pile 13M - Filtered Version

This is a filtered version of timaeus/dsir-pile-13m.

Filtering Applied:

—Excluded: All rows where metadata.pile_set_name contains 'Github' or 'DM_mathematics'
—Kept: All other rows from the original dataset

Dataset Size

—Original: ~13M examples
—Filtered: 12,782,200 examples (99.9% of original)
—Uploaded in: 64 batch files

Usage

python

from datasets import load_dataset
dataset = load_dataset("timaeus/dsir-pile-13m-filtered-no-github-or-dm_mathematics")

Technical Details

—Dataset uploaded in Parquet format for efficiency
—Each batch contains up to 200,000 examples
—Memory-efficient processing used during creation

Filtering Logic

Rows were excluded if their metadata.pile_set_name list contained:

—'Github'
—'DM_mathematics'