gist-sparse-attention/GSA-PT-Qwen2-7B-Instruct-chunk8-data
GSA-PT-Qwen2-7B-Instruct-chunk8-data This is the continue pretraining dataset used for training GSA (Gist Sparse Attention) models based on Qwen2-7B-Instruct with chunk size chunk8. Each sample is tokenized and formatted with GSA gist tokens for continue pretraining. Paper GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding Related Model yuzhenm/GSA-PT-Qwen2-7B-Instruct-chunk8 — model trained on this dataset
GSA-PT-Qwen2-7B-Instruct-chunk8-data
This is the continue pretraining dataset used for training GSA (Gist Sparse Attention) models based on Qwen2-7B-Instruct with chunk size chunk8.
Dataset Description
- Source: Filtered from FineWeb (sample-10BT), keeping samples with sequence lengths ranging from 2K to 4K tokens.
- Chunk size:
chunk8
Each sample is tokenized and formatted with GSA gist tokens for continue pretraining.
Paper
GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding
Related Model
- yuzhenm/GSA-PT-Qwen2-7B-Instruct-chunk8 — model trained on this dataset