gist-sparse-attention/GSA-PT-Llama-3.2-1B-chunk8-data
GSA-PT-Llama-3.2-1B-chunk8-data This is the continue pretraining dataset used for training GSA (Gist Sparse Attention) models with chunk size chunk8. Each sample is tokenized and formatted with GSA gist tokens for continued pretraining. Paper GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding Related Models yuzhenm/GSA-PT-Llama-3.2-1B-chunk8 — model trained on this dataset
0255
GSA-PT-Llama-3.2-1B-chunk8-data
This is the continue pretraining dataset used for training GSA (Gist Sparse Attention) models with chunk size chunk8.
Dataset Description
- Source: Filtered from FineWeb (sample-10BT)
- Filtering: Samples with sequence lengths ranging from 2K to 4K tokens
- Chunk size:
chunk8 - Splits:
train,eval
Each sample is tokenized and formatted with GSA gist tokens for continued pretraining.
Paper
GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding
Related Models
- yuzhenm/GSA-PT-Llama-3.2-1B-chunk8 — model trained on this dataset