gist-sparse-attention/GSA-PT-Qwen2-7B-Instruct-chunk4-chunk4

GSA-PT-Qwen2-7B-Instruct-chunk4-chunk4

This model is continued pretrained from Qwen/Qwen2-7B-Instruct using Gist Sparse Attention (GSA) with chunk size chunk4-chunk4.

Paper

GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding

Model Details