OutrageouslyBad200/SMAT_ablations

1---2license: mit3library_name: pytorch4tags:5  - language-modeling6  - transformer7  - attention8  - ablation9  - research10language:11  - en12datasets:13  - HuggingFaceFW/fineweb-edu14metrics:15  - perplexity16pipeline_tag: text-generation17---18 19# SMAT — Semantic Attention20 21Trained checkpoints for **SMAT** (Semantic Attention), a transformer attention22variant with a learnable semantic-similarity bias and per-token value gate.23 24- **Code:** [github.com/OutrageouslyBad200/smat](https://github.com/OutrageouslyBad200/smat)25- **Architecture:** 24 layers × 384d × 6 heads, block size 256, ~64 M parameters26- **Tokenizer:** GPT-2 (`tiktoken`, vocab 50 257)27- **Training data:** FineWeb-Edu sample-10BT, 98 M tokens28- **Training compute:** 12 000 optimizer steps, batch 16 × grad_accum 2 (effective 32),29  RTX 406030 31## Equation32 33```34Attn(Q,K,V) = softmax(QK^T/sqrt(d_k) + λ·S + P + M) · (G ⊙ V)35```36 37- `S_ij = cos(W_s h_i, W_s h_j)` — cosine similarity in shared projection38- `c_j = (1/n) Σ_{l≤j} S_jl` — causal semantic centrality39- `G_j = σ(w_g^T h_j + μ·c_j + β)` — per-token value gate40- `λ = softplus(λ_raw)` — constrained positive scalar (per layer)41 42## Repository contents43 44This HuggingFace repo hosts 20 checkpoints from the 5-seed ablation in45Experiment 6 of the SMAT paper:46 47```48baseline_s0/final.pt   s_only_s0/final.pt   g_only_s0/final.pt   full_s0/final.pt49baseline_s1/final.pt   s_only_s1/final.pt   g_only_s1/final.pt   full_s1/final.pt50baseline_s2/final.pt   s_only_s2/final.pt   g_only_s2/final.pt   full_s2/final.pt51baseline_s3/final.pt   s_only_s3/final.pt   g_only_s3/final.pt   full_s3/final.pt52baseline_s4/final.pt   s_only_s4/final.pt   g_only_s4/final.pt   full_s4/final.pt53```54 55Each variant directory also contains `config.json` and `metrics.jsonl`56(per-step training + eval logs).57 58| Variant | `use_S` | `use_G` | Description |59|---------|---------|---------|-------------|60| `baseline` | False | False | Standard attention |61| `s_only`   | True  | False | Semantic bias only |62| `g_only`   | False | True  | Value gate only |63| `full`     | True  | True  | Full SMAT |64 65## Results66 67Validation perplexity on FineWeb-Edu, 5 seeds, 12 000 steps:68 69| Variant | Mean ppl | Std | Δ vs baseline | Seed wins |70|---------|----------|-----|---------------|-----------|71| Baseline | 79.75 | 1.69 | — | — |72| S-only | 79.47 | 1.71 | −0.35% | 4/5 |73| G-only | 79.02 | 1.65 | −0.90% | 5/5 |74| **Full SMAT** | **78.65** | 1.75 | **−1.37%** | **5/5** |75 760 NaN failures across 240 000 optimizer steps.77 78## Usage79 80```bash81pip install torch numpy tiktoken huggingface_hub82git clone https://github.com/OutrageouslyBad200/smat.git83cd smat84```85 86Download a single checkpoint:87 88```python89from huggingface_hub import hf_hub_download90ckpt_path = hf_hub_download(91    repo_id="OutrageouslyBad200/smat",92    filename="full_s0/final.pt",93)94```95 96Load it into the SMAT model:97 98```python99import torch100from model import Config, SMATTransformer101 102state = torch.load(ckpt_path, map_location="cuda")103cfg = Config(**state["config"])104model = SMATTransformer(cfg).cuda()105model.load_state_dict(state["state_dict"])106model.eval()107```108 109Reproduce surgical ablations (Experiment 7):110 111```bash112python ablate.py --ckpt full_s0/final.pt --n_batches 80113```114 115## Surgical-ablation findings (Experiment 7)116 117Run on Full SMAT, val ppl 79.010:118 119| Ablation | val ppl | Δ |120|----------|---------|---|121| λ=0 (S still drives c) | 79.40 | +0.49% |122| S removed entirely | 80.48 | +1.85% |123| Random S (same norm) | 81.23 | +2.80% |124| G replaced by mean | 196.99 | +149% |125| G forced to 1.0 | 625 850 | catastrophic |126 127- The gate **G** is catastrophically essential.128- **S** routes mostly through `μ·c` in the gate (74 % of lift), not through129  `λ·S` in attention (26 %).130- Per-token gate differentiation matters: replacing G with its mean costs 149 %.131 132## Limitations133 134- Small base model (~64 M params); larger-scale runs (100 M on FineWeb / FineMath)135  show stronger lifts (−11 % to −17 %) but are not included as released checkpoints.136- Trained only on English FineWeb-Edu sample-10BT — generalization to other137  domains untested at this scale.138- Not instruction-tuned, not RLHF'd, no safety filtering. Research artifact only.139 140## Citation141 142```bibtex143@misc{smat2026,144  author       = {OutrageouslyBad200},145  title        = {SMAT: Semantic Attention},146  year         = {2026},147  howpublished = {\url{https://github.com/OutrageouslyBad200/smat}},148}149```150 151## Contact152 153For further information on training runs, intermediate experiments, or the154unpublished paper draft, please contact the creator via155[GitHub](https://github.com/OutrageouslyBad200) or HuggingFace.156 157## License158 159[MIT License](https://github.com/OutrageouslyBad200/smat/blob/main/LICENSE).160