CoolFace
Modelpublic

gist-sparse-attention/GSA-PT-Llama-3.2-1B-chunk4-chunk4

sourceHugging Faceupdated 6mo agoView on Hugging Face
0likes33downloads
Model Card

GSA-PT-Llama-3.2-1B-chunk4-chunk4

This model is continued pretrained from meta-llama/Llama-3.2-1B using Gist Sparse Attention (GSA) with chunk size chunk4-chunk4.

Paper

GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding

Model Details

FieldValue
Base modelmeta-llama/Llama-3.2-1B
Training typeContinued Pretraining
Chunk sizechunk4-chunk4
ArchitectureLlama-3.2-1B