PY007/slimpajama_llama_tokenized_upsample_4096_chunk_1M

Generated using https://github.com/FranxYao/Long-Context-Data-Engineering with the below command: mkdir logs mkdir data mkdir data/slimpajama mkdir data/slimpajama/per_source_downsample cd data_engineering PATH_TO_SLIMPAJAMA=rokset3/slim_pajama_chunk_1 nohup python -u slimpajama_packing.py\ --dataset_size=5b\ --print_interval=100 --num_process=200\ --chunk_size=1000001 \ --dataset_path=$PATH_TO_SLIMPAJAMA\ --output_path=../data/slimpajama/per_source_downsample/… See the full description on the dataset page: https://huggingface.co/datasets/PY007/slimpajama_llama_tokenized_upsample_4096_chunk_1M.