datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
owt-sampleThese files were created with the following script:
from datasets import load_dataset
from tqdm import tqdm
import io
dataset = load_dataset("Skylion007/openwebtext")['train']
split_dataset = dataset.train_test_split(train_size=2400000, test_size=60000, seed=0)
with io.open('data/owt_train.txt','w') as fopen:
listout = []
for data in tqdm(split_dataset['train']):
listout.append(data['text']+'<|endoftext|>')
if len(listout) > 1000:
_ =… See the full description on the dataset page: https://huggingface.co/datasets/stanford-cs336/owt-sample.cs336-basics-collection
CS336 Assignment 1 — Pre-tokenized Data & BPE Tokenizers
This repository contains preprocessing artifacts produced for Stanford CS336: Language Modeling from Scratch, Spring 2025 — Assignment 1: Basics.
It includes:
pre-tokenized TinyStories train/validation data,
pre-tokenized OpenWebText (OWT sample) train/validation data,
byte-level BPE vocabularies and merge tables for both datasets.
The main purpose of this repository is to avoid repeating the relatively expensive… See the full description on the dataset page: https://huggingface.co/datasets/victorhu493/cs336-basics-collection.