fhai50032/BiBo_corpus-4096-packed-qtk-2.05M-hi35-en65
fhai50032/BiBo_corpus-4096-packed-qtk-2.05M-hi35-en65 Packed pretraining corpus, 2,050,000 rows x 4096 tokens = 8.397B tokens, tokenized with fhai50032/QTK-81K. Format column type notes label list<int32> exactly 4096 tokens, no padding raw_label string label decoded back to text (redundant, for inspection) There is no attention_mask column: the corpus is packed, so every position is a real token and the mask would be all ones on every row.… See the full description on the dataset page: https://huggingface.co/datasets/fhai50032/BiBo_corpus-4096-packed-qtk-2.05M-hi35-en65.
fhai50032/BiBo_corpus-4096-packed-qtk-2.05M-hi35-en65
Packed pretraining corpus, 2,050,000 rows x 4096 tokens = 8.397B tokens, tokenized with fhai50032/QTK-81K.
Format
There is no `attention_mask` column: the corpus is packed, so every position is a real token and the mask would be all ones on every row.
How it was built
Documents are tokenized and appended to a running buffer with a single <|im_end|> (id 81914) between them. Every time the buffer reaches 4096 tokens a row is emitted and the remainder carries into the next row. So no row is padded, no document is truncated, and a document longer than 4096 tokens simply spans rows.
Source: /home/marimo/work/data/packed_mix.
Important: do not mask id 0
QTK-81K has pad_token_id == eos_token_id == 0. The separator here is <|im_end|> (81914), not <|endoftext|>, so id 0 never appears in this corpus. Masking it is therefore a no-op and is safe, but it is also unnecessary -- there is no padding to mask.
If you adapt this recipe to a corpus that separates documents with <|endoftext|>, masking id 0 would delete every document-boundary target and the model would never learn to stop -- a silent failure that reads as degeneration rather than a bug.