stanford-cs336/owt-sample

These files were created with the following script: from datasets import load_dataset from tqdm import tqdm import io dataset = load_dataset("Skylion007/openwebtext")['train'] split_dataset = dataset.train_test_split(train_size=2400000, test_size=60000, seed=0) with io.open('data/owt_train.txt','w') as fopen: listout = [] for data in tqdm(split_dataset['train']): listout.append(data['text']+'<|endoftext|>') if len(listout) > 1000: _ =… See the full description on the dataset page: https://huggingface.co/datasets/stanford-cs336/owt-sample.