nkkbr/Mini-K3-1H-attnres-full-sublayer-v1_D

1# Mini-K3-1H-attnres-full-sublayer2 3Independent, text-only Mini-K3-1H architecture package for the AttnRes ablation.4This package is self-contained: copy this directory alone to define the model,5construct it on the meta device, or randomly initialize it. No pretrained6weights are bundled or required.7 8**Residual mechanism:** Full Attention Residuals, attention/FFN sublayer granularity. The embedding, every attention output A_i, and every feed-forward output F_i remain separate sources. Before each sublayer, a learned pseudo-query softmax mixes all sources produced so far. The final mix sees 27 sources. This is finer than block size 1.9 10All other dimensions match Mini-K3-1H: 163,840 vocabulary entries, hidden width111,024, 13 decoder layers (9 KDA and 4 NoPE Gated MLA), 64 routed experts with12top-4 selection, Stable LatentMoE, online-QB-compatible router state, SiTU-GLU,13and untied input/output embeddings. Logical parameter count: 1,016,780,524; report-style14active count per token: 353,556,204.15 16## Files17 18- `config.json` and `configuration_mini_k3.py`: complete configuration and validation.19- `modeling_mini_k3.py`: differentiable full model, including this residual variant.20- `tokenizer.py`, `tiktoken.model`, `tokenizer_config.json`: matching K3 tokenizer.21- `parameter_schema.jsonl`: every parameter name, shape, dtype and gradient flag.22- `initialize_model.py`: explicit meta construction and optional random initialization.23- `verify_config.py`, `smoke_test.py`: CPU/meta-only structural and forward/backward tests.24- `ARCHITECTURE.md`, `VARIANT.md`, `TRAINING.md`, `PROVENANCE.md`: model and integration notes.25- `MANIFEST.sha256`: integrity checksums for all shipped files.26 27## Verify without using a GPU28 29```bash30python3 verify_config.py31python3 smoke_test.py32python3 initialize_model.py33```34 35The default `initialize_model.py` command stays on the meta device and does not36allocate full-size weights. `--materialize` is explicit and optional.37