fengluoqiuwu/owt-bucket

OWT GPT-2 sentence segments (buckets 256 / 512 / 1024 / 2048)

Tokenized sentence/paragraph segments of OpenWebText (Skylion007/openwebtext, revision main), encoded with GPT-2 BPE plus three extra special tokens, then assigned to length buckets.

This repository is a derived dataset. It does not rediscover or replace the original text corpus. Every input_ids sequence comes from documents in OpenWebText.

• —Packaging / conversion code & this card: MIT (see LICENSE)
• —Underlying web text: inherited from OpenWebText / Skylion007/openwebtext. Use of the token sequences remains subject to the original corpus terms and to the licenses of the crawled pages.

中文摘要

本仓库是 OpenWebText 的派生数据：用 GPT-2 BPE 按句段切分（process_d=2048），再按有效长度分到 256/512/1024/2048 四档。validation 对应本地训练里的 dev。内容与 bdelf 本地缓存 owt_owt-bucket_2dba74e39372313f（fingerprint 2dba74e39372313f）逐 token 一致。仓库协议为 MIT；文本来源仍归 OpenWebText。

本地训练存盘时每行会再 pad 到 2048 以便 memmap。Hub 上只保留到有效 length（pad 可按 PAD_ID 确定性还原）。

What's in a row

Bucket rule (same as training):

• —128 ≤ n ≤ 256 → 256
• —n ≤ 512 → 512
• —n ≤ 1024 → 1024
• —n ≤ 2048 → 2048

To restore the exact bdelf training row (storage width 2048, right-padded with PAD_ID):

PAD_ID = 50257
row = example["input_ids"] + [PAD_ID] * (2048 - example["length"])

Training loss in bdelf masks with length, not with bucket. bucket is the pad width used when mixing similar-length samples.

Splits

OpenWebText on Hub only ships a train split. Document-level holdout is applied on parquet row order of Skylion007/openwebtext (identical to `fengluoqiuwu/owt-seg512`):

Holdout: shuffle indices 0 .. N-1 with holdout_seed=42, take first 1,024 rows → test documents, next 1,024 → dev documents.

Train bucket counts

Validation / test bucket counts are in meta.json.

Tokenization

Each segment is wrapped as [BOS] + content + [EOS]. Segments with wrapped length ≤ 128 are dropped (min_chunk_len=128).

Segmentation (must match training)

Documents are not concatenated. For each document:

1. 1.GPT-2 Fast encode with offset_mapping (byte BPE; punctuation is matched on UTF-8 text, then mapped to token boundaries).
2. 2.Content cap d - 2 = 2046, process_d=2048.
3. 3.If remaining tokens exceed the cap, look back min(256, d/8)=256 tokens and cut on the best delimiter: paragraph blank lines (P0) → single newline (P1) → sentence punctuation (P2). If none, hard-cut at the cap and continue from that index (no tokens dropped).
4. 4.Assign a bucket, then (in bdelf) pad to 2048 for memmap storage. This Hub export strips pad.

Block shuffle after cutting: shuffle_seed=42, block size 65,536 rows.

Fingerprint of the local cache this export was taken from: `2dba74e39372313f`.

Load

from datasets import load_dataset

ds = load_dataset("fengluoqiuwu/owt-bucket")
ex = ds["train"][0]
print(len(ex["input_ids"]), ex["length"], ex["bucket"])

# only the 512-bucket
ds512 = ds["train"].filter(lambda r: r["bucket"] == 512)

Streaming:

ds = load_dataset("fengluoqiuwu/owt-bucket", split="train", streaming=True)

Relation to the sibling dataset

`fengluoqiuwu/owt-seg512` uses process_d=512 and a single pad width 512. Do not mix owt-seg512 with this repo's 512-bucket in the same step: cut points differ, so the 512-length populations are not the same.

Source / citation

Primary source: Skylion007/openwebtext.

@misc{Gokaslan2019OpenWeb,
  title={OpenWebText Corpus},
  author={Aaron Gokaslan and Vanya Cohen},
  year={2019},
  howpublished={\url{https://skylion007.github.io/OpenWebTextCorpus/}}
}

Please also cite this derived segmentation if you use these exact splits, cuts, and buckets.

License

MIT for the files in this Hub repository (conversion, metadata, token-id tables). The original OpenWebText documents remain a scrape of public web pages; this repo does not grant additional rights to that underlying text.