Datasetpublic

BEE-spoke-data/wikipedia-20230901.en-deduped

wikipedia - 20230901.en - deduped purpose: train with less data while maintaining (most) of the quality This is really more of a "high quality diverse sample" rather than "we are trying to remove literal duplicate documents". Source dataset: graelo/wikipedia. configs default command: python -m text_dedup.minhash \ --path $ds_name \ --name $dataset_config \ --split $data_split \ --cache_dir "./cache" \ --output $out_dir \ --column… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/wikipedia-20230901.en-deduped.

sourceHugging Facecc-by-sa-3.0updated 9mo agoView on Hugging Face

6likes1.4kdownloads

Dataset Card

wikipedia - 20230901.en - deduped

purpose: train with less data while maintaining (most) of the quality

This is really more of a "high quality diverse sample" rather than "we are trying to remove literal duplicate documents". Source dataset: graelo/wikipedia.

configs

`default`

command:

python -m text_dedup.minhash \
  --path $ds_name \
  --name $dataset_config \
  --split $data_split \
  --cache_dir "./cache" \
  --output $out_dir \
  --column $text_column \
  --ngram 4 --threshold 0.6 \
  --hash_func xxh3 --hash_bits 16 --num_perm 64 \
  --batch_size 10000

dedup:

Fingerprinting... (num_proc=40): 100% 6705754/6705754 [06:57<00:00, 16063.27 examples/s]
Iterating MinHashes...: 100% 671/671 [04:13<00:00,  2.65it/s]
Clustering...: 100% 10/10 [00:21<00:00,  2.18s/it]
Finding clusters... (num_proc=40): 100% 6705754/6705754 [06:38<00:00, 16839.42 examples/s]
Filtering clusters... (num_proc=40): 100% 6705754/6705754 [02:25<00:00, 46058.39 examples/s]
Saving the dataset (39/39 shards): 100% 5971972/5971972 [03:47<00:00, 26266.10 examples/s]
[10/23/23 02:29:41] INFO     Loading                         : 78.82s

result:

python

DatasetDict({
    train: Dataset({
        features: ['id', 'url', 'title', 'text'],
        num_rows: 5673373
    })
    validation: Dataset({
        features: ['id', 'url', 'title', 'text'],
        num_rows: 149299
    })
    test: Dataset({
        features: ['id', 'url', 'title', 'text'],
        num_rows: 149300
    })
})

text-only

This is the same thing but with all columns except for 'text' removed.

python

from datasets import load_dataset

# If the dataset is gated/private, make sure you have run huggingface-cli login
config_name = "text-only"
dataset = load_dataset("BEE-spoke-data/wikipedia-deduped", config_name)

token counts

train

Using tiktoken GPT-4 tokenizer, train split, and text column:

	num_tokens
count	5.67337e+06
mean	612.413
std	739.331
min	3
25%	163
50%	359
75%	761
max	34298

total: 3,474,446,396