datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
owt-sampleThese files were created with the following script:
from datasets import load_dataset
from tqdm import tqdm
import io
dataset = load_dataset("Skylion007/openwebtext")['train']
split_dataset = dataset.train_test_split(train_size=2400000, test_size=60000, seed=0)
with io.open('data/owt_train.txt','w') as fopen:
listout = []
for data in tqdm(split_dataset['train']):
listout.append(data['text']+'<|endoftext|>')
if len(listout) > 1000:
_ =… See the full description on the dataset page: https://huggingface.co/datasets/stanford-cs336/owt-sample.owt-processed_128eng_spa_owt_phonemized_fullVRI_Underwater_Graspingowt-processed_512eng_spa_owt_word_fullowt-processed_32owt-gpt2-1024-splitowt-processed_64cs336-owt-32k-bpe
CS336 OWT 32k BPE tokenized
Pre-tokenized OpenWebText sample (Stanford CS336 assignment 1 source) using a custom 32k byte-level BPE tokenizer.
Files
owt_vocab_32k.pkl — vocab dict[int, bytes] (32000 entries; ID 0 = <|endoftext|>, 1..256 = raw bytes, 257..31999 = merges)
owt_merges_32k.pkl — list[tuple[bytes, bytes]] in merge order
owt_train.npy — uint16 token IDs of 11 GB owt_train.txt (2,729,574,028 tokens)
owt_valid.npy — uint16 token IDs of 277 MB owt_valid.txt (66… See the full description on the dataset page: https://huggingface.co/datasets/AndrewK101/cs336-owt-32k-bpe.owtThese files were created with the following script:
from datasets import load_dataset
from tqdm import tqdm
import io
dataset = load_dataset("Skylion007/openwebtext")['train']
split_dataset = dataset.train_test_split(train_size=2400000, test_size=60000, seed=0)
with io.open('data/owt_train.txt','w') as fopen:
listout = []
for data in tqdm(split_dataset['train']):
listout.append(data['text']+'<|endoftext|>')
if len(listout) > 1000:
_ =… See the full description on the dataset page: https://huggingface.co/datasets/soumitrapy/owt.owt-processed_256owt-bucket
OWT GPT-2 sentence segments (buckets 256 / 512 / 1024 / 2048)
Tokenized sentence/paragraph segments of OpenWebText (Skylion007/openwebtext, revision main), encoded with GPT-2 BPE plus three extra special tokens, then assigned to length buckets.
This repository is a derived dataset. It does not rediscover or replace the original text corpus. Every input_ids sequence comes from documents in OpenWebText.
Packaging / conversion code & this card: MIT (see LICENSE)
Underlying web… See the full description on the dataset page: https://huggingface.co/datasets/fengluoqiuwu/owt-bucket.eng_spa_owt_phonemized_full_romanizedlaces-owt-rwkv-tokens
LACES training data: OpenWebText RWKV tokens (512)
Preprocessed OpenWebText token shards for training LACES (Linear-Attention Continuous-state DiffuSion) — the DiffRWKV State-Hijacking RELAY 2.9B 512-token trajectory champion recipe.
304,052 samples, max length 512, tokenized with the RWKV7-Goose-World3 tokenizer
Format: one *_tokens.npz per sample
Usage
hf download humanlong/laces-owt-rwkv-tokens --repo-type dataset --local-dir .
tar xzf… See the full description on the dataset page: https://huggingface.co/datasets/humanlong/laces-owt-rwkv-tokens.owt_and_hh_rlhf
Dataset Card for "owt_and_hh_rlhf"
More Information needed
owt-ngrams
Dataset Card for OpenWebText n-grams
Dataset Summary
This dataset contains 246K of the most common token-based (GPT-2/GPT-3) n-grams (n=1 to n=6), in the OpenWebText (OWT) dataset.
For convenient searching, it provides full tokens/strings, as well as per-position tokens/strings.
Usage
Generally, this dataset allows identifying the most common n-grams in a text corpus.
When researching LLMs tokenized similarly to GPT-2/GPT-3, it allows:
Constructing… See the full description on the dataset page: https://huggingface.co/datasets/danwil/owt-ngrams.owt2details_nthngdy__pythia-owt2-70m-100k
Dataset Card for Evaluation run of nthngdy/pythia-owt2-70m-100k
Dataset Summary
Dataset automatically created during the evaluation run of model nthngdy/pythia-owt2-70m-100k on the Open LLM Leaderboard.
The dataset is composed of 64 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 2 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train"… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard-old/details_nthngdy__pythia-owt2-70m-100k.owt-intitial-tokensdetails_nthngdy__pythia-owt2-70m-50k
Dataset Card for Evaluation run of nthngdy/pythia-owt2-70m-50k
Dataset Summary
Dataset automatically created during the evaluation run of model nthngdy/pythia-owt2-70m-50k on the Open LLM Leaderboard.
The dataset is composed of 64 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 2 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train"… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard-old/details_nthngdy__pythia-owt2-70m-50k.chess_owtowt_bpeowt-seg512
OWT GPT-2 sentence segments (fixed pad 512)
Tokenized sentence/paragraph segments of OpenWebText (Skylion007/openwebtext, revision main), encoded with GPT-2 BPE plus three extra special tokens.
This repository is a derived dataset. It does not rediscover or replace the original text corpus. Every input_ids sequence comes from documents in OpenWebText.
Packaging / conversion code & this card: MIT (see LICENSE)
Underlying web text: inherited from OpenWebText /… See the full description on the dataset page: https://huggingface.co/datasets/fengluoqiuwu/owt-seg512.tokenized-owtdllm-dev-owtowt-ngrams
Dataset Card for OpenWebText n-grams
Dataset Summary
This dataset contains 246K of the most common token-based (GPT-2/GPT-3) n-grams (n=1 to n=6), in the OpenWebText (OWT) dataset.
For convenient searching, it provides full tokens/strings, as well as per-position tokens/strings.
Usage
Generally, this dataset allows identifying the most common n-grams in a text corpus.
When researching LLMs tokenized similarly to GPT-2/GPT-3, it allows:
Constructing… See the full description on the dataset page: https://huggingface.co/datasets/Yuki5177/owt-ngrams.owt-gpt2bpe-local-9BOpenWebText BPE bins used by the 400M replication runs (branch fixedinit-embed-init). train.bin ~9.03B tokens, val.bin ~4M tokens. Checksums in PROVENANCE.md5.
owt-processed_16owt-processed_8