gist-sparse-attention/GSA-PT-Qwen2-7B-Instruct-chunk16

GSA-PT-Qwen2-7B-Instruct-chunk16

This model is continued pretrained from Qwen/Qwen2-7B-Instruct using Gist Sparse Attention (GSA) with chunk size chunk16.

Paper

GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding

Model Details