Datasetpublic

fengluoqiuwu/owt-bucket

OWT GPT-2 sentence segments (buckets 256 / 512 / 1024 / 2048) Tokenized sentence/paragraph segments of OpenWebText (Skylion007/openwebtext, revision main), encoded with GPT-2 BPE plus three extra special tokens, then assigned to length buckets. This repository is a derived dataset. It does not rediscover or replace the original text corpus. Every input_ids sequence comes from documents in OpenWebText. Packaging / conversion code & this card: MIT (see LICENSE) Underlying web… See the full description on the dataset page: https://huggingface.co/datasets/fengluoqiuwu/owt-bucket.

sourceHugging Facemitupdated 1mo agoView on Hugging Face

0likes71downloads

settings

This repository belongs to fengluoqiuwu on Hugging Face.

CoolFace never edits a repository it does not host. Visibility, licence, collaborators and gating are all managed at the source.

nameowt-bucket

visibilitypublic

licencemit

gatedno

ownerfengluoqiuwu

Account settings