Datasetpublic

eoinf/wikitext_gptneox

Dataset Card for eoinf/wikitext_gptneox Original dataset Original dataset: Salesforce/wikitext Dataset Details Total Tokens: 122,236,928 Total Sequences: 119,372 Context Length: 1024 tokens Tokenizer: EleutherAI/gpt-neox-20b Format: Each example contains a single field tokens with a list of 1024 token IDs Preprocessing Each document was: Tokenized using the EleutherAI/gpt-neox-20b tokenizer Prefixed with a BOS (beginning of… See the full description on the dataset page: https://huggingface.co/datasets/eoinf/wikitext_gptneox.

sourceHugging Facemitupdated 11mo agoView on Hugging Face

0likes5downloads

Dataset Card

Dataset Card for eoinf/wikitext_gptneox

Original dataset

Original dataset: Salesforce/wikitext

Dataset Details

—Total Tokens: 122,236,928
—Total Sequences: 119,372
—Context Length: 1024 tokens
—Tokenizer: EleutherAI/gpt-neox-20b
—Format: Each example contains a single field tokens with a list of 1024 token IDs

Preprocessing

Each document was:

1.Tokenized using the EleutherAI/gpt-neox-20b tokenizer
2.Prefixed with a BOS (beginning of sequence) token
3.Suffixed with an EOS (end of sequence) token
4.Packed into fixed-length sequences of 1024 tokens

Usage

python

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("eoinf/wikitext_gptneox")

# Access training data
train_data = dataset["train"]
print(train_data[0]["tokens"])  # First sequence

Use with PyTorch

python

import torch
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("eoinf/wikitext_gptneox", split="train")

# Convert to PyTorch tensors
dataset.set_format(type="torch", columns=["tokens"])

# Create DataLoader
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

for batch in dataloader:
    tokens = batch["tokens"]