OutrageouslyBad200/SMAT_ablations

1---2license: mit3library_name: pytorch4tags:5  - language-modeling6  - transformer7  - attention8  - ablation9  - research10language:11  - en12datasets:13  - HuggingFaceFW/fineweb-edu14metrics:15  - perplexity16pipeline_tag: text-generation17---18# SMAT — Semantic Attention19 20Trained checkpoints for **SMAT** (Semantic Attention), a transformer attention21variant with a learnable semantic-similarity bias and per-token value gate.22 23- **Code:** [github.com/OutrageouslyBad200/SMATest](https://github.com/OutrageouslyBad200/SMATest)24- **Architecture:** 24 layers × 384d × 6 heads, block size 256, ~64 M parameters25- **Tokenizer:** GPT-2 (`tiktoken`, vocab 50 257)26- **Training data:** FineWeb-Edu sample-10BT, 98 M tokens27- **Training compute:** 12 000 optimizer steps, batch 16 × grad_accum 2 (effective 32),28  RTX 406029## Equation30```31Attn(Q,K,V) = softmax(QK^T/sqrt(d_k) + λ·S + P + M) · (G ⊙ V)32```33- `S_ij = cos(W_s h_i, W_s h_j)` — cosine similarity in shared projection34- `c_j = (1/n) Σ_{l≤j} S_jl` — causal semantic centrality35- `G_j = σ(w_g^T h_j + μ·c_j + β)` — per-token value gate36- `λ = softplus(λ_raw)` — constrained positive scalar (per layer)37## Repository contents38This HuggingFace repo hosts 20 checkpoints from the 5-seed ablation in39Experiment 6 of the SMAT paper:40```41baseline_s0/final.pt   s_only_s0/final.pt   g_only_s0/final.pt   full_s0/final.pt42baseline_s1/final.pt   s_only_s1/final.pt   g_only_s1/final.pt   full_s1/final.pt43baseline_s2/final.pt   s_only_s2/final.pt   g_only_s2/final.pt   full_s2/final.pt44baseline_s3/final.pt   s_only_s3/final.pt   g_only_s3/final.pt   full_s3/final.pt45baseline_s4/final.pt   s_only_s4/final.pt   g_only_s4/final.pt   full_s4/final.pt46```47Each variant directory also contains `config.json` and `metrics.jsonl`48(per-step training + eval logs).49| Variant | `use_S` | `use_G` | Description |50|---------|---------|---------|-------------|51| `baseline` | False | False | Standard attention |52| `s_only`   | True  | False | Semantic bias only |53| `g_only`   | False | True  | Value gate only |54| `full`     | True  | True  | Full SMAT |55## Results56Validation perplexity on FineWeb-Edu, 5 seeds, 12 000 steps:57| Variant | Mean ppl | Std | Δ vs baseline | Seed wins |58|---------|----------|-----|---------------|-----------|59| Baseline | 79.75 | 1.69 | — | — |60| S-only | 79.47 | 1.71 | −0.35% | 4/5 |61| G-only | 79.02 | 1.65 | −0.90% | 5/5 |62| **Full SMAT** | **78.65** | 1.75 | **−1.37%** | **5/5** |630 NaN failures across 240 000 optimizer steps.64## Usage65```bash66pip install torch numpy tiktoken huggingface_hub67git clone https://github.com/OutrageouslyBad200/SMATest.git68cd SMATest69```70Download a single checkpoint:71```python72from huggingface_hub import hf_hub_download73ckpt_path = hf_hub_download(74    repo_id="OutrageouslyBad200/SMATest",75    filename="full_s0/final.pt",76)77```78Load it into the SMAT model:79 80```python81import torch82from model import Config, SMATTransformer83state = torch.load(ckpt_path, map_location="cuda")84cfg = Config(**state["config"])85model = SMATTransformer(cfg).cuda()86model.load_state_dict(state["state_dict"])87model.eval()88```89 90Reproduce surgical ablations (Experiment 7):91 92```bash93python ablate.py --ckpt full_s0/final.pt --n_batches 8094```95 96## Surgical-ablation findings (Experiment 7)97 98Run on Full SMAT, val ppl 79.010:99 100| Ablation | val ppl | Δ |101|----------|---------|---|102| λ=0 (S still drives c) | 79.40 | +0.49% |103| S removed entirely | 80.48 | +1.85% |104| Random S (same norm) | 81.23 | +2.80% |105| G replaced by mean | 196.99 | +149% |106| G forced to 1.0 | 625 850 | catastrophic |107 108- The gate **G** is catastrophically essential.109- **S** routes mostly through `μ·c` in the gate (74 % of lift), not through110  `λ·S` in attention (26 %).111- Per-token gate differentiation matters: replacing G with its mean costs 149 %.112 113## Limitations114 115- Small base model (~64 M params); larger-scale runs (100 M on FineWeb / FineMath)116  show stronger lifts (−11 % to −17 %) but are not included as released checkpoints.117- Trained only on English FineWeb-Edu sample-10BT — generalization to other118  domains untested at this scale.119- Not instruction-tuned, not RLHF'd, no safety filtering. Research artifact only.120 121## Citation122 123```bibtex124@misc{smat2026,125  author       = {OutrageouslyBad200},126  title        = {SMAT: Semantic Attention},127  year         = {2026},128  howpublished = {\url{https://github.com/OutrageouslyBad200/SMATest}},129}130```131 132## Contact133 134For further information on training runs, intermediate experiments, or the135unpublished paper draft, please contact the creator via136[GitHub](https://github.com/OutrageouslyBad200) or HuggingFace.137 138## License139 140[MIT License](https://github.com/OutrageouslyBad200/smat/blob/main/LICENSE).141