sparse_llama
neuralmagic_-_SparseLlama-3-8B-pruned_50.2of4-ggufSparse-Llama-3.1-8B-2of4-GGUFSparseLLM_-_prosparse-llama-2-7b-ggufllama2.c-stories42M-gsm8k-sparse-only-uncompressedSparseLLM_-_prosparse-llama-2-13b-ggufSparse-Llama-3.1-8B-2of4-GGUFZhMax_-_llama-2-7b-ebft-sparsegpt-outlier-wiki-block-outlier-ggufllama2.c-stories42M-gsm8k-sparse-only-compressed
GSA-PT-Llama-3.2-1B-chunk8-data
GSA-PT-Llama-3.2-1B-chunk8-data
This is the continue pretraining dataset used for training GSA (Gist Sparse Attention) models with chunk size chunk8.
Each sample is tokenized and formatted with GSA gist tokens for continued pretraining.
Paper
GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding
Related Models
yuzhenm/GSA-PT-Llama-3.2-1B-chunk8 — model trained on this dataset
GSA-PT-Llama-3.2-1B-chunk16-data
GSA-PT-Llama-3.2-1B-chunk16-data
This is the continue pretraining dataset used for training GSA (Gist Sparse Attention) models with chunk size chunk16.
Each sample is tokenized and formatted with GSA gist tokens for continued pretraining.
Paper
GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding
Related Models
yuzhenm/GSA-PT-Llama-3.2-1B-chunk16 — model trained on this dataset
GSA-PT-Llama-3.2-1B-chunk4-chunk4-data
GSA-PT-Llama-3.2-1B-chunk4-chunk4-data
This is the continue pretraining dataset used for training GSA (Gist Sparse Attention) models with chunk size chunk4-chunk4.
Each sample is tokenized and formatted with GSA gist tokens for continued pretraining.
Paper
GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding
Related Models
yuzhenm/GSA-PT-Llama-3.2-1B-chunk4-chunk4 — model trained on this dataset
GSA-FT-Llama-3.2-1B-data
GSA-FT-Llama-3.2-1B-data
This is the supervised fine-tuning dataset used for training GSA (Gist Sparse Attention) models based on Llama-3.2-1B.
The answers to the questiosn are generated by Deepseek-V3.2
Each sample is formatted with GSA gist tokens for instruction fine-tuning.
Paper
GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding
Related Models
yuzhenm/GSA-FT-Llama-3.2-1B-chunk8
yuzhenm/GSA-FT-Llama-3.2-1B-chunk16… See the full description on the dataset page: https://huggingface.co/datasets/gist-sparse-attention/GSA-FT-Llama-3.2-1B-data.