Datasetpublic

AndrewK101/cs336-owt-32k-bpe

CS336 OWT 32k BPE tokenized Pre-tokenized OpenWebText sample (Stanford CS336 assignment 1 source) using a custom 32k byte-level BPE tokenizer. Files owt_vocab_32k.pkl — vocab dict[int, bytes] (32000 entries; ID 0 = <|endoftext|>, 1..256 = raw bytes, 257..31999 = merges) owt_merges_32k.pkl — list[tuple[bytes, bytes]] in merge order owt_train.npy — uint16 token IDs of 11 GB owt_train.txt (2,729,574,028 tokens) owt_valid.npy — uint16 token IDs of 277 MB… See the full description on the dataset page: https://huggingface.co/datasets/AndrewK101/cs336-owt-32k-bpe.

sourceHugging Facemitupdated 5mo agoView on Hugging Face

1likes156downloads

Dataset Card

CS336 OWT 32k BPE tokenized

Pre-tokenized OpenWebText sample (Stanford CS336 assignment 1 source) using a custom 32k byte-level BPE tokenizer.

Files

—owt_vocab_32k.pkl — vocab dict\[int, bytes\] (32000 entries; ID 0 = <|endoftext|>, 1..256 = raw bytes, 257..31999 = merges)
—owt_merges_32k.pkl — list\[tuple[bytes, bytes]\] in merge order
—owt_train.npy — uint16 token IDs of 11 GB owt_train.txt (2,729,574,028 tokens)
—owt_valid.npy — uint16 token IDs of 277 MB owt_valid.txt (66,458,625 tokens)

ID scheme matches cs336_basics.tokenizer.Tokenizer (special at 0).

Quickstart on RunPod

pip install -U huggingface_hub
huggingface-cli download AndrewK101/cs336-owt-32k-bpe     --repo-type dataset --local-dir data