OutrageouslyBad200/SMAT_ablations
0
1================================================================================2SMAT (Semantic Attention) - Findings Summary for Paper3================================================================================4Compiled: 2026-06-035Source files used (all in this repo, all re-verified for this document):6 runs/{baseline,s_only,g_only,full}_s{0..4}/metrics.jsonl (20 training runs)7 runs_4ksteps/{baseline,s_only,g_only,full}_s0/metrics.jsonl (4 partial runs)8 figures_final/ablation_inference.json (surgical ablations)9 figures_final/ablation_results.csv (final eval table)10 11Every number in sections 2-6 below was recomputed directly from the raw12metrics.jsonl files (not copied from any pre-existing CSV or README). The13per-seed deltas, means, std devs, and lambda/G statistics were all14re-derived from scratch and cross-checked twice.15 16The cross-scale results in section 7 are from prior experiments outside17this repo; they are cited as such and were not re-verified from raw data18here. Treat them as historical context, not as primary evidence.19 20 21================================================================================221. WHAT SMAT IS (one paragraph)23================================================================================24SMAT modifies standard transformer attention with two additions: (1) a25learnable semantic-similarity bias S added to the QK^T logits, scaled by a26per-layer positive scalar lambda; and (2) a per-token value gate G that27multiplies V before attention aggregation. The gate is also influenced by28a causal semantic-centrality term c derived from S. Equation:29 30 Attn(Q,K,V) = softmax( QK^T / sqrt(d_k) + lambda * S + P + M ) . (G (*) V)31 32where33 S_ij = cos(W_s h_i, W_s h_j) shared-projection cosine similarity34 c_j = (1/n) sum_{l<=j} S_jl causal semantic centrality35 G_j = sigmoid(w_g^T h_j + mu*c_j + b) per-token value gate36 lambda = softplus(lambda_raw) constrained positive, per layer37 P learnable [L,L] positional bias38 M causal mask39 40W_s is shared across heads within a layer. G is per-token and broadcast41across head channels. Four variants were trained:42 baseline use_S = False, use_G = False (standard attention)43 s_only use_S = True, use_G = False (lambda*S only)44 g_only use_S = False, use_G = True (gate only)45 full use_S = True, use_G = True (both components)46 47 48================================================================================492. TRAINING SETUP (the 5-seed ablation)50================================================================================51Architecture: 24 layers, 384 dim, 6 heads, block size 256, vocab 5025752Parameters: 63,376,512 (~64M) per variant53Optimizer: Fused AdamW, lr 6e-4, 200-step linear warmup, cosine to 6e-554Batch: 16, grad_accum 2 -> effective batch 3255Tokens per step: 16 * 2 * 256 = 8,19256Total steps: 12,000 -> 98,304,000 tokens per run (~98M)57Eval: every 500 steps on 40 val batches (val_ppl = exp(val_loss))58Precision: bf16 autocast, TF32, cudnn.benchmark, SDPA attention path59Hardware: single RTX 4060 (vram ~6 GB during training)60Corpus: FineWeb-Edu sample-10BT, GPT-2 tokenization61Seeds: {0, 1, 2, 3, 4} for each of the four variants -> 20 runs total62 63Every variant uses an IDENTICAL data stream for the same seed (data_seed=133764plus run seed); this is what makes the per-seed comparison meaningful.65 66 67================================================================================683. PRIMARY RESULT - 5-SEED ABLATION TABLE69================================================================================70Final val perplexity at step 11999 (98M tokens seen), per variant per seed:71 72 seed baseline s_only g_only full73 ---- --------- ---------------- ---------------- ----------------74 0 78.3244 78.7377 (+0.53%) 78.2175 (-0.14%) 77.8634 (-0.59%)75 1 80.9320 80.8649 (-0.08%) 80.5771 (-0.44%) 80.0262 (-1.12%)76 2 82.3365 82.0735 (-0.32%) 81.3363 (-1.21%) 81.2602 (-1.31%)77 3 79.3884 77.9830 (-1.77%) 77.9834 (-1.77%) 77.7016 (-2.12%)78 4 77.7449 77.6807 (-0.08%) 77.0092 (-0.95%) 76.3953 (-1.74%)79 80Per-seed deltas are computed as (variant_ppl - baseline_ppl) / baseline_ppl81within the same seed, NOT against the baseline mean.82 83Aggregate (mean across 5 seeds; std is population std):84 85 variant n mean_ppl std delta_mean seed wins vs baseline86 -------- -- --------- ------- ------------ -----------------------87 Baseline 5 79.7452 1.6894 - -88 S-only 5 79.4680 1.7131 -0.348% 4 / 589 G-only 5 79.0247 1.6463 -0.904% 5 / 590 Full SMAT 5 78.6493 1.7498 -1.374% 5 / 591 92Key observations:93 - Full SMAT beats baseline on EVERY seed (5/5).94 - The two components are additive in direction: S contributes ~0.35%,95 G contributes ~0.90%, and Full SMAT (both together) gives ~1.37%.96 S + G individually sum to ~1.25%, slightly less than full's 1.37%;97 the components are mostly additive, with a small interaction in their98 favor.99 - Variance ACROSS seeds (std ~1.7 ppl) is larger than the mean effect100 (~1.1 ppl). The signal is real but small relative to seed noise; it101 only becomes clear because every variant uses the SAME data stream per102 seed, so the within-seed comparison cancels almost all of that noise.103 - The smallest within-seed gain for Full SMAT is -0.59% (seed 0);104 the largest is -2.12% (seed 3). Direction is consistent; magnitude105 varies by roughly a factor of 4 across seeds.106 107 108================================================================================1094. DIAGNOSTICS (training dynamics)110================================================================================1114.1 Lambda (semantic-bias scale, per-layer, softplus-positive)112 Init value (all layers, all seeds): 0.1269 ( = softplus(lambda_raw_init) )113 End-of-training mean lambda (averaged across 24 layers), per seed:114 Full SMAT: s0 0.3069 s1 0.3075 s2 0.3031 s3 0.2954 s4 0.3176115 -> mean across seeds ~ 0.306116 S-only: s0 0.3140 s1 0.3188 s2 0.3221 s3 0.3183 s4 0.3261117 -> mean across seeds ~ 0.320118 119 Per-layer pattern (Full SMAT seed 0 final checkpoint, 24 layers):120 layers 0-2: lambda ~ 0.14 - 0.18 (slightly below init)121 layers 3-8: lambda ~ 0.08 - 0.16 (DECREASES below init)122 layers 9-17: lambda ~ 0.32 - 0.68 (climbs sharply)123 layers 18-23: lambda ~ 0.18 - 0.58 (settles, still elevated)124 global min: 0.0834 (layer 5)125 global max: 0.6821 (layer 16)126 127 Across all 10 SMAT-bearing runs (s_only + full, 5 seeds each),128 the observed per-layer lambda range was [0.059, 0.715]. The shape -129 suppressed in early layers, dominant in middle layers - is robust130 across all 5 seeds.131 1324.2 Gate G (per-token, sigmoid output, averaged over tokens for logging)133 Initial G_mean per run is around 0.5 (sigmoid initialized near zero134 pre-activation). Across training G_mean rises briefly toward ~0.58135 in the first ~1000 steps, then decays.136 Final G_mean at step 11999 (Full SMAT): s0 0.3506, s1 0.3333,137 s2 0.3306, s3 0.3447, s4 0.3397 -> mean ~ 0.340138 Final G_mean (G-only): s0 0.3484, s1 0.3359,139 s2 0.3676, s3 0.3625, s4 0.3563 -> mean ~ 0.354140 G_std (across tokens) at end of training is ~ 0.16, meaning the gate141 is NOT collapsed: many tokens have G well above the mean, many have it142 well below. Per-token differentiation is preserved.143 1444.3 Stability145 NaN failures across all 20 runs at 12k steps each (= 240,000 optimizer146 steps total): ZERO.147 No NaN-driven step skips were recorded in any run. Training is148 numerically clean under bf16 + TF32 + fused AdamW on this hardware.149 1504.4 Wall-clock cost151 Baseline runs: ~4000 - 5300 sec per run (~ 1.1 - 1.5 hours)152 S-only runs: ~4700 - 6100 sec153 G-only runs: ~4300 - 5700 sec154 Full runs: ~5400 - 6600 sec (~ 1.5 - 1.8 hours)155 Cost overhead of Full SMAT vs baseline: roughly +20 to +35% wall time156 at this size on a 4060. (Variation within each variant is hardware/157 thermal noise across the days the runs were executed.)158 159 160================================================================================1615. INFERENCE-TIME SURGICAL ABLATIONS162================================================================================163Performed on the Full SMAT seed-0 final checkpoint (no retraining; the164trained weights are loaded, individual components are zeroed/replaced165during a forward pass, and val ppl is re-measured on 80 val batches).166Source: figures_final/ablation_inference.json167 168Intact Full SMAT (this measurement, 80 batches): val ppl = 79.010169(The 40-batch eval at the end of training reported 77.863; the 80-batch170slice used for the ablation is a different val subset, hence the small171difference. What matters is the delta within this measurement context.)172 173 ablation val_ppl delta vs intact what was changed174 ------------ ----------- ----------------- ------------------------175 lambda_zero 79.40 +0.49% lambda set to 0 in the176 attention bias only;177 S still flows through178 mu*c into the gate.179 S_off 80.48 +1.85% S removed completely:180 no lambda*S in attention181 AND no mu*c in the gate.182 random_S 81.22 +2.80% S replaced by a random183 matrix with the same184 Frobenius norm. Tests185 whether S's STRUCTURE186 matters, or just its norm.187 G_one 625,850 +792,016% G forced to constant 1.0188 (gate disabled).189 G_mean 196.99 +149.32% G replaced by its scalar190 mean over tokens (per-191 token differentiation192 removed; mean value kept).193 194Decomposition of S's contribution:195 Total S contribution (S_off vs intact): 1.85%196 Contribution via lambda*S in attention: 0.49% -> 26% of S's lift197 Contribution via mu*c in the gate: 1.36% -> 74% of S's lift198 Most of S's value is being read through the gate's centrality term,199 NOT through the additive logit bias.200 201Per-layer lambda=0 ablations (24 separate ablations, zeroing lambda in202one layer at a time):203 Maximum single-layer cost: +0.0580% (layer 14)204 Sum of all 24 per-layer deltas: +0.219%205 This is far less than the +0.49% you get from zeroing lambda in every206 layer at once. Interpretation: the lambda*S contribution is REDUNDANT207 across layers - no single layer is load-bearing. The benefit is small208 and distributed.209 210Structural vs scalar test:211 random_S (+2.80%) hurts MORE than S_off (+1.85%). Replacing S with a212 same-norm random matrix is worse than removing it entirely. This means213 the model is using S's learned STRUCTURE, not just the presence of an214 extra signal of a given magnitude. A random matrix with the right215 energy actively damages the model; trained S is genuinely informative.216 217Gate criticality:218 G=1.0 collapses the model entirely (val ppl > 600,000). The gate is not219 a small modulation - the trained network depends on it as a primary220 routing mechanism.221 G=mean costs +149%. Replacing per-token gating with a single scalar mean222 destroys most of the gate's value. The gate's worth is in WHICH tokens223 it suppresses or amplifies, not in its average level.224 225 226================================================================================2276. PARTIAL 4K-STEP RUN (1 seed, 32M tokens) - CONTEXT ONLY228================================================================================229A shorter run at 4,000 steps (32,768,000 tokens) was performed for ONE230seed before the full 12k-step ablation was launched. It is not directly231comparable to the 5-seed result above (different budget, no seed232variance), but it shows the same direction and a larger relative gap233because perplexity is higher overall at lower training budgets.234 235 variant val_ppl delta vs baseline236 --------- --------- ------------------237 baseline 148.4449 -238 s_only 147.8903 -0.37%239 g_only 144.9733 -2.34%240 full 141.5651 -4.63%241 242This is consistent with the broader pattern: at smaller training budgets243(or smaller scales), the relative effect of SMAT is larger. As training244proceeds and the baseline gets closer to its asymptote, the absolute245ppl gap and the relative gap both shrink. The 12k-step 5-seed result246(section 3) is the rigorous number; the 4k-step result is a directional247sanity check.248 249DO NOT cite the 4k-step delta as if it were comparable to the 12k-step250delta. The user has already flagged this as a "partial run, not directly251comparable" in the repo README.252 253 254================================================================================2557. CROSS-SCALE CONTEXT (PRIOR EXPERIMENTS, NOT FROM THIS REPO)256================================================================================257The following are described in this repo's README as prior experiments258done before this codebase. They are NOT re-verified from raw data here259and should be cited as historical context, not as primary evidence of260this paper's findings. If the paper depends on them, the underlying logs261should be located and re-verified before they appear in any final claim.262 263 scale corpus tokens delta seeds note264 ----- ------------------- -------- ------- ----- -------------------265 7M Shakespeare short mixed 1 directional crossover266 around step 150267 100M FineWeb (smoke) short -7% 1 100-step smoke run268 100M FineWeb 100M -11% 3 of 3 strongest replicated269 100M FineMath 100M -17% 3 of 3 strongest result270 68M FineWeb (SMAT+A90) 20M -22.5% 1 SMAT combined with271 BlockAttnRes; out272 of scope for V1.273 64M FineWeb (THIS REPO) 98M -1.37% 5 of 5 primary result274 275The pattern across these (taking them at face value): direction is276preserved across every scale, every corpus, every seed where data277exists. Magnitude shrinks as the scale and training budget grow, with278the exception of the 100M experiments where the gap is largest.279 280If the paper's headline number is from the 100M FineWeb or FineMath281experiments, the V1 evidence is the 5-seed run in section 3 of THIS282document plus those prior experiments. The 5-seed result is the283hardest-to-dispute piece because every variant trained on identical284data and direction is consistent across 5/5 seeds.285 286 287================================================================================2888. WHAT'S LOAD-BEARING vs WHAT IS NOT289================================================================================290Load-bearing (do not remove):291 - The gate G. Catastrophic if removed (+792,016%) and badly degraded292 if reduced to its mean (+149%). The gate is the single most important293 component SMAT adds.294 - The structure of S (not just its norm). random_S hurts more than295 S_off, so the learned S projection contains real information.296 - The mu*c centrality channel from S into the gate. This accounts for297 ~74% of S's total contribution. If S is used at all, it should be298 used HERE.299 300Not load-bearing in any individual layer (and largely redundant overall):301 - The per-layer lambda*S term in the attention bias. Removing it in302 one layer costs at most 0.06% ppl. Removing it in all layers costs303 only 0.49%. The benefit exists but it is small and distributed; no304 single layer's lambda is critical, and a uniform lambda across305 layers is the wrong abstraction (early layers actively prefer it306 near zero, middle layers want it near 0.5-0.7).307 308Net implication for V2 (this is what the data argues for):309 - Drop the per-layer lambda_raw parameters and the lambda*S term in310 attention. Keep S's projection W_s and keep mu*c flowing into the311 gate. You retain ~74% of S's lift while removing 24 scalar312 parameters and one matrix add per layer.313 - If you want lambda to stay, replace per-layer lambda with a314 per-layer gate-mix scalar driven by the same per-layer pattern315 observed here (suppressed in early layers, elevated in middle316 layers).317 318 319================================================================================3209. CAVEATS, LIMITATIONS, THINGS NOT TO OVERCLAIM321================================================================================322 - The headline -1.37% at 64M / 98M tokens is real (5/5 seeds, identical323 per-seed data streams; one-sided sign test gives p = 1/32 = 0.03125324 under the null hypothesis of no effect), but it is SMALL in absolute325 terms (about 1.1 perplexity points). The within-seed effect is what326 makes it visible; do not present mean +/- std as if seed noise were327 the only consideration.328 - Std across seeds (1.65 - 1.75) is larger than the mean effect.329 Always quote the WITHIN-SEED comparison or the SIGN TEST when making330 the case, not just delta-of-means with overlapping error bars.331 - The surgical ablation results in section 5 are from ONE checkpoint332 (Full SMAT seed 0). The qualitative conclusions (G is critical, S333 routes through mu*c, layer-wise lambda is redundant) are stark334 enough that one checkpoint is informative, but it would be more335 defensible to repeat on at least one more seed before publication.336 - The cross-scale numbers in section 7 are not verified from raw data337 in this repo. Do not put them in a results table without separately338 locating and confirming the underlying logs.339 - All training was on a single RTX 4060 with bf16 + TF32. Numbers may340 shift slightly on different hardware; the direction should not.341 - Tokenizer is GPT-2 byte-pair. Corpus is FineWeb-Edu sample-10BT.342 Results on other corpora (FineMath in particular, see section 7)343 may differ in magnitude.344 345 346================================================================================34710. SUGGESTED PAPER CLAIMS (ordered by strength of evidence in this repo)348================================================================================349STRONG (directly supported by 20-run ablation, this repo, this document):350 1. Adding a per-token value gate to attention reduces val ppl by351 ~0.9% on FineWeb at 64M params, 5/5 seeds.352 2. Adding both the semantic-similarity bias AND the gate reduces353 val ppl by ~1.4% on FineWeb at 64M params, 5/5 seeds.354 3. Training is numerically stable (0 NaN failures over 240,000355 optimizer steps across 20 runs).356 4. lambda exhibits a robust per-layer pattern: suppressed in early357 layers, elevated in middle layers, settled in late layers.358 This pattern is consistent across all 5 seeds.359 360STRONG (supported by surgical-ablation evidence, single checkpoint):361 5. S's contribution flows ~74% through the gate's centrality term362 and only ~26% through the additive logit bias.363 6. The trained S has learned STRUCTURE; replacing it with a random364 matrix of equal norm hurts more than removing it entirely.365 7. The per-token gate is catastrophic to remove and very expensive366 to replace with a mean. Per-token differentiation matters.367 368WEAKER (need cross-scale / cross-seed re-verification before paper):369 8. SMAT's relative effect is larger at smaller training budgets and370 at certain corpora (FineMath in particular).371 9. SMAT directionality is preserved across scales 7M to 100M.372 373DO NOT CLAIM without further work:374 - That SMAT is a uniformly large improvement (it is not at this scale).375 - That SMAT helps at scales beyond 100M (no data yet).376 - That per-layer lambda is essential (the surgical ablation shows it377 is largely redundant).378 379 380================================================================================38111. RAW NUMBERS (machine-readable, for any tables in the paper)382================================================================================38312k-step 5-seed ablation, final val ppl:384 385baseline = [78.32438343443285, 80.93203870134798, 82.33645607970243,386 79.38843805716219, 77.74486565469935]387s_only = [78.73771250646860, 80.86492907631813, 82.07353893301419,388 77.98296438017636, 77.68072112072338]389g_only = [78.21752726763506, 80.57706364462581, 81.33629330240751,390 77.98339201086276, 77.00922636947199]391full = [77.86344065821494, 80.02616085611275, 81.26017429181104,392 77.70156877103959, 76.39525023929815]393 394Means:395 baseline 79.74523638546896396 s_only 79.46797320334013397 g_only 79.02470051900063398 full 78.64931896329529399 400Population std:401 baseline 1.6894... s_only 1.7131... g_only 1.6463... full 1.7498...402 403Sample (n-1) std (if you prefer):404 baseline 1.8888 s_only 1.9153 g_only 1.8406 full 1.9563405 406Delta of means vs baseline (%):407 s_only -0.3477 g_only -0.9035 full -1.3743408 409Per-layer trained lambda (Full SMAT seed 0, 24 values, from410ablation_inference.json):411 [0.1837, 0.1491, 0.1557, 0.1365, 0.1209, 0.0834, 0.1213, 0.0920,412 0.1605, 0.3192, 0.3323, 0.4566, 0.3248, 0.5507, 0.5484, 0.4943,413 0.6821, 0.4693, 0.3914, 0.5843, 0.1961, 0.3536, 0.2781, 0.1833]414 (printed to 4 dp; full precision in figures_final/ablation_inference.json)415 416Surgical ablation deltas (Full SMAT seed 0, 80-batch eval slice, intact417ppl 79.010):418 lambda_zero +0.4903% S_off +1.8547% random_S +2.8032%419 G_one +792,015.9609% G_mean +149.3211%420 421 422================================================================================423END424================================================================================425