Datasetpublic

pszemraj/simplepile-lite

Dataset Card for "simplepile-lite" Interleaved dataset using 'first exhausted' strategy. Counts: DatasetDict({ train: Dataset({ features: ['text'], num_rows: 452432 }) validation: Dataset({ features: ['text'], num_rows: 1000 }) test: Dataset({ features: ['text'], num_rows: 11908 }) }) token counts - train using GPTNeoX Tokenizer: token_count count 452432 mean 868.642 std… See the full description on the dataset page: https://huggingface.co/datasets/pszemraj/simplepile-lite.

sourceHugging Faceapache-2.0updated 9mo agoView on Hugging Face

1likes97downloads

Dataset Card

Dataset Card for "simplepile-lite"

Interleaved dataset using 'first exhausted' strategy. Counts:

python

DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 452432
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 1000
    })
    test: Dataset({
        features: ['text'],
        num_rows: 11908
    })
})

token counts - train

using GPTNeoX Tokenizer:

	token_count
count	452432
mean	868.642
std	4791.71
min	3
25%	88
50%	232
75%	590
max	1.39747e+06