Datasetpublic

datajuicer/redpajama-pile-stackexchange-refined-by-data-juicer

RedPajama & The Pile -- StackExchange (refined by Data-Juicer) A refined version of StackExchange dataset in RedPajama & The Pile by Data-Juicer. Removing some "bad" samples from the original merged dataset to make it higher-quality. This dataset is usually used to pretrain a Large Language Model. Notice: Here is a small subset for previewing. The whole dataset is available here (About 71GB). Dataset Information Number of samples: 26,309,203 (Keep ~57.89% from… See the full description on the dataset page: https://huggingface.co/datasets/datajuicer/redpajama-pile-stackexchange-refined-by-data-juicer.

sourceHugging Faceapache-2.0updated 3y agoView on Hugging Face

0likes21downloads

Dataset Card

RedPajama & The Pile -- StackExchange (refined by Data-Juicer)

A refined version of StackExchange dataset in RedPajama & The Pile by Data-Juicer. Removing some "bad" samples from the original merged dataset to make it higher-quality.

This dataset is usually used to pretrain a Large Language Model.

Notice: Here is a small subset for previewing. The whole dataset is available here (About 71GB).

Dataset Information

—Number of samples: 26,309,203 (Keep ~57.89% from the original dataset)

Refining Recipe

yaml

# global parameters
project_name: 'Data-Juicer-stack-exchange'
dataset_path: '/path/to/your/dataset'  # path to your dataset directory or file
export_path: '/path/to/your/dataset.jsonl'

np: 50  # number of subprocess to process your dataset
open_tracer: true

# process schedule
# a list of several process operators with their arguments
process:
  - clean_email_mapper:
  - clean_links_mapper:
  - fix_unicode_mapper:
  - punctuation_normalization_mapper:
  - whitespace_normalization_mapper:

  - alphanumeric_filter:
      tokenization: false
      min_ratio: 0.35  # <3sigma
      max_ratio: 0.943  # 3sigma
  - average_line_length_filter:  # for code
      min_len: 20  # >3sigma
      max_len: 400 # >3sigma
  - character_repetition_filter:
      rep_len: 10
      max_ratio: 0.4  # >3sigma (0.12)
  - flagged_words_filter:
      lang: en
      tokenization: true
      max_ratio: 0.01  # >3sigma
  - language_id_score_filter:  # remove language filter
      min_score: 0.1 # <3sigma
  - maximum_line_length_filter:  # for code
      min_len: 80
  - perplexity_filter:
      lang: en
      max_ppl: 10000  # >3sigma
  - special_characters_filter:
      min_ratio: 0.232 # 3sigma
      max_ratio: 0.7  # >3sigma
  - text_length_filter:
      min_len: 200
  - words_num_filter:
      lang: en
      tokenization: true
      min_num: 100
  - word_repetition_filter:
      lang: en
      tokenization: true
      rep_len: 10
      max_ratio: 0.8 # >3sigma

  - document_simhash_deduplicator:  #26309203 left
      tokenization: space
      window_size: 3
      lowercase: true
      ignore_pattern: '\n\n'
      num_blocks: 9
      hamming_distance: 7