dsir
Datasets
All datasets matching “dsir”dsir-pile-10kdsir-pile-13m-filtered-no-github-or-dm_mathematics
DSIR Pile 13M - Filtered Version
This is a filtered version of timaeus/dsir-pile-13m.
Filtering Applied:
Excluded: All rows where metadata.pile_set_name contains 'Github' or 'DM_mathematics'
Kept: All other rows from the original dataset
Dataset Size
Original: ~13M examples
Filtered: 12,782,200 examples (99.9% of original)
Uploaded in: 64 batch files
Usage
from datasets import load_dataset
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/timaeus/dsir-pile-13m-filtered-no-github-or-dm_mathematics.dsir-pile-10m-tokensDSIR-filtered-pile-50M
Dataset Card for DSIR-filtered-pile-50M
Dataset Summary
This dataset is a subset of The Pile, selected via the DSIR data selection method. The target distribution for DSIR is the Wikipedia and BookCorpus2 subsets of The Pile.
Languages
English (EN)
Dataset Structure
A train set is provided (51.2M examples) in jsonl format.
Data Instances
{"contents": "Hundreds of soul music enthusiasts from the United Kingdom plan to make their way to… See the full description on the dataset page: https://huggingface.co/datasets/stanford-crfm/DSIR-filtered-pile-50M.dsir-pile-10k-tokensds_iros_3tasks