Kiy-K/icml2026-spon-paper127-artifacts

Public Trackio logbook: https://huggingface.co/spaces/Kiy-K/icml2026-spon-paper127-trackio Hub Collection: https://huggingface.co/collections/Kiy-K/icml-2026-repro-spon-paper-127-6a58806b89e9fee9bb3b72a1

ICML 2026 paper #127 — SPON reproduction report

Paper: Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneity (OpenReview 4v5iCXWmcR). This is an independent, budget-capped reproduction on the paper-supported mistralai/Mistral-7B-Instruct-v0.2 backbone. It is not a Gemma architecture-transfer result.

Executive summary

Claim 1 is supported: the 224 input-independent SPON vectors trained with the backbone frozen and folded with a 0.0 BF16 maximum final-logit difference, leaving no additional SPON matrix multiplication. Claim 2 is supported: SPON block perplexity was 24.172 versus TEAL 25.479, and lm-eval word perplexity was 10.693 versus 10.774. Claim 3 remains inconclusive under a limited subset, Claim 4 was not tested without a compatible sparse kernel, and Claim 5 is supported. Total estimated Jobs spend was $12.17, below the $20 ceiling.

Primary WikiText result

Dataset: Salesforce/wikitext, wikitext-2-raw-v1, revision b08601e04326c79dfdd32d625aee71d232d685c3. Raw text is tokenized without a chat template and concatenated into non-overlapping blocks of 128 tokens. The full test split contains 4,358 raw rows, 330,303 raw tokenizer tokens, 2,580 complete blocks, 330,240 retained tokens, and 327,660 next-token predictions. There are zero exact train/test block-hash matches.

The block metric sums shifted-token cross-entropy over every complete test block, divides once by the exact prediction count, and exponentiates once. The separate lm-eval==0.4.9 run is included because its WikiText word-perplexity metric is closer to the paper's reporting convention. Both comparisons reuse the same in-memory checkpoint and tokenizer across dense, TEAL, and SPON.

Mechanism and foldability

All 224 q/k/v/o and gate/up/down projections across 32 Mistral decoder layers receive exact per-token top-k input masking and one trainable, input-independent vector. At 50% sparsity the implemented equation is

\[ Y = W S(X) + W\alpha. \]

Only 1,245,184 SPON scalars train, or 0.01719456% of the 7,241,732,096 checkpoint parameters. Every alpha received a finite non-zero gradient in the GPU smoke. After training, W alpha is stored as an explicit frozen projection bias. The BF16 full-model fold test reported max and mean final-logit differences of 0.0 and 0.0, within the explicit tolerance 0.25. Before folding there are 224 additional SPON vector-matrix multiplications per model forward. After folding there are 224 fixed bias additions and 1,376,256 fixed bias scalars, but no additional SPON matrix multiplication after bias folding.

Calibration

• —Model revision: 63a8b081895390a26e140280378bc85ec8bce07a
• —Official source commit: 6fe499180c0010c266fd9713b45910b726485777
• —Seed: 1337
• —BF16, block size 128, 10 epochs
• —AdamW, learning rate 1e-5, weight decay 0, linear decay, no warmup
• —Microbatch 8, gradient accumulation 6, effective batch 48
• —Frozen pretrained weights; train SPON only
• —Objective: released Mistral parallel dense-teacher/sparse-student KL plus rescaled layerwise projection MSE
• —Calibration set: 36,718 rows, 2,774,656 retained tokens, 21,677 blocks
• —Hardware/runtime: one NVIDIA A100-SXM4-80GB, Python 3.12, PyTorch 2.13.0+cu130, CUDA 13.0; complete dependency pins are in pyproject.toml
• —Deviation: one-GPU accumulation approximates the released six-GPU batch; the dynamic loss rescaling is applied per microbatch and is not mathematically identical to DDP

Representation analysis

On 2,048 identical token representations, SPON had higher CKA in 31/32 layers (mean 0.998688 vs. TEAL 0.996087) and lower relative L2 in 31/32 (mean 0.080928 vs. TEAL 0.104388). Claim 5 verdict: Supported.

The evaluator computes exact centered linear CKA in sample-space Gram form, global relative Frobenius L2 distance, and mean tokenwise cosine similarity for every decoder layer. Identical token IDs and layers are used for dense, TEAL, and SPON. Raw per-layer values are in results/representation_results.json.

Downstream subset

The evaluator pins lm-eval==0.4.9, zero-shot prompts, no chat template, max length 4096, batch size 2, and a limit of 100 per task/subtask. Exact task names are commonsense_qa, mathqa, medmcqa, mmlu, openbookqa, and truthfulqa_mc1. Because mmlu expands to 57 subtasks, its limit applies to each subtask. The complete legacy MathQA loader was reviewed and pinned at revision c4f1cc784c04c4957b50c97858f23893b633eea6 before enabling its required remote-code flag narrowly for math_qa.

This limited run is explicitly not described as a full six-benchmark replication.

Throughput boundary

The official magnitude-sort/mask implementation still dispatches dense torch.nn.functional.linear operations. A dense PyTorch mask is not evidence of sparse inference acceleration. Therefore no wall-clock dense-vs-TEAL speedup is reported, and Claim 4 is not tested. Folding does remove SPON's separate correction multiplication, but that mechanism result must not be conflated with sparse-kernel throughput.

Implementation audits and deviations

1. 1.The released Mistral forward references cache locals when cache is disabled; full-sequence runs retain use_cache=True.
2. 2.The released set_bias() result is not consumed by the active column_masking() path; this reproduction routes the correction through the actual projection path.
3. 3.The released BF16 order W(S(X)+alpha) produced a 4.765625 final-logit difference after algebraic folding. The reproduction executes the paper equation literally as W S(X) + W alpha, then stores the exact same correction as the folded bias.
4. 4.Raw block perplexity and lm-eval word perplexity are both reported instead of silently choosing the metric closest to the paper's numbers.
5. 5.The released Datasets pipeline concatenates and drops remainders separately inside each mapping batch. This reproduction concatenates the same per-row, special-token-bearing tokenizer outputs across the full split and drops one final remainder. Exact retained-token counts are reported, and every condition sees the identical resulting blocks.
6. 6.Failed, canceled, diagnostic, and abandoned prior-scope jobs remain visible but do not contribute evidence to the Mistral verdicts.
7. 7.A repeat seed and 60% calibration were not launched: either requires a second roughly 3.34-hour full calibration and would consume the reserve needed for full evaluation and publication under the $20 ceiling.

Compute and artifacts

Estimated Hugging Face Jobs spend was $12.17 against the $20 cap: $10.49 for the Mistral reproduction and $1.68 for the earlier abandoned Gemma scope. Costs use recorded running seconds and public hourly flavor prices; canceled Jobs without a returned finished timestamp use the documented estimate.

Every reported value is traceable to JSON under results/. Source, configuration, environment, failures, Posterly source/PDF/PNG, and the Trackio logbook bundle are published in the companion Hugging Face dataset and collection.