trytryw/fixed-alpha-scale-pilot-v1

Fixed Alpha Score-Scale Pilot v1

Public artifacts for a controlled 125M-parameter, 4K-context language-model study of fixed attention score scaling with full sparsemax (alpha=2), full entmax-1.5, and a softmax reference.

Layout

• —metadata/: immutable execution records, plans, logs, source/config snapshots, and SHA-256 manifests.
• —runs/<run-name>/: complete run backups, including configuration, provenance, metrics, final training state, and preregistered analysis checkpoints.
• —diagnostics/: compressed offline attention diagnostics and their manifests.
• —results/: machine-readable aggregate results.

All 15 trajectories use seed 0, 1,280 steps, and 20,971,520 training tokens per run. Eleven belong to the original matrix and four to the preregistered lower-scale boundary extension.

Main result

The combined tuned gap is 0.03779 NLL, down 69.92% from 0.12560 at the default scale. Both selected scales are now interior points: c=0.5 for entmax-1.5 and c=0.125 for sparsemax. This remains a single-seed exploratory study whose selection and reporting use the same validation split.

Verification

Each run directory contains backup-manifest.json. Downloaded files should be checked against that manifest before use. The original matrix used repository commit d2fdb0538adec47a95728ce93481aaff56ec19a5; the extension used 69e2a5df64844184a2a144964922b747e039938f. Both have the same frozen training-source hash.

These are exploratory single-seed research artifacts, not production-ready pretrained models. Backend runtime differences must not be attributed to alpha alone because sparsemax and entmax-1.5 use different AdaSplash implementations.