Datasetpublic

PY007/slimpajama_llama_tokenized_upsample_4096_chunk_1M

Generated using https://github.com/FranxYao/Long-Context-Data-Engineering with the below command: mkdir logs mkdir data mkdir data/slimpajama mkdir data/slimpajama/per_source_downsample cd data_engineering PATH_TO_SLIMPAJAMA=rokset3/slim_pajama_chunk_1 nohup python -u slimpajama_packing.py\ --dataset_size=5b\ --print_interval=100 --num_process=200\ --chunk_size=1000001 \ --dataset_path=$PATH_TO_SLIMPAJAMA\ --output_path=../data/slimpajama/per_source_downsample/… See the full description on the dataset page: https://huggingface.co/datasets/PY007/slimpajama_llama_tokenized_upsample_4096_chunk_1M.

sourceHugging Faceupdated 2y agoView on Hugging Face

2likes146downloads

Dataset Card

Generated using https://github.com/FranxYao/Long-Context-Data-Engineering with the below command:

bash

mkdir logs
mkdir data
mkdir data/slimpajama
mkdir data/slimpajama/per_source_downsample
cd data_engineering

PATH_TO_SLIMPAJAMA=rokset3/slim_pajama_chunk_1
nohup python -u slimpajama_packing.py\
    --dataset_size=5b\
    --print_interval=100 --num_process=200\
    --chunk_size=1000001 \
    --dataset_path=$PATH_TO_SLIMPAJAMA\
    --output_path=../data/slimpajama/per_source_downsample/ --down_sample_ratio=0.1 --down_sample_mode=per_source\
    > ../logs/slimpajama_packing_dist_per_source_downsample_0.1.log 2>&1 &
tail -f ../logs/slimpajama_packing_dist_per_source_downsample_0.1.log