gist-sparse-attention/GSA-PT-Llama-3.2-1B-chunk16-data

GSA-PT-Llama-3.2-1B-chunk16-data This is the continue pretraining dataset used for training GSA (Gist Sparse Attention) models with chunk size chunk16. Each sample is tokenized and formatted with GSA gist tokens for continued pretraining. Paper GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding Related Models yuzhenm/GSA-PT-Llama-3.2-1B-chunk16 — model trained on this dataset