Datasetpublic

AndrewK101/cs336-owt-32k-bpe

CS336 OWT 32k BPE tokenized Pre-tokenized OpenWebText sample (Stanford CS336 assignment 1 source) using a custom 32k byte-level BPE tokenizer. Files owt_vocab_32k.pkl — vocab dict[int, bytes] (32000 entries; ID 0 = <|endoftext|>, 1..256 = raw bytes, 257..31999 = merges) owt_merges_32k.pkl — list[tuple[bytes, bytes]] in merge order owt_train.npy — uint16 token IDs of 11 GB owt_train.txt (2,729,574,028 tokens) owt_valid.npy — uint16 token IDs of 277 MB… See the full description on the dataset page: https://huggingface.co/datasets/AndrewK101/cs336-owt-32k-bpe.

sourceHugging Facemitupdated 5mo agoView on Hugging Face

1likes175downloads

settings

This repository belongs to AndrewK101 on Hugging Face.

CoolFace never edits a repository it does not host. Visibility, licence, collaborators and gating are all managed at the source.

namecs336-owt-32k-bpe

visibilitypublic

licencemit

gatedno

ownerAndrewK101

Account settings