juiceb0xc0de/nova-1-1.3b-code-atlas

juiceb0xc0de/nova-1-1.3b-code-atlas

A brain atlas for [Smilyai-labs/Nova-1-Standard-1.3B-Preview](https://huggingface.co/Smilyai-labs/Nova-1-Standard-1.3B-Preview), a 24-layer Mixture-of-Depths transformer. This is not a chat dataset or a benchmark. It is an internal-mechanics map built by running activations through a corpus of prompts and scoring what each layer, component, head, and feature direction is doing.

If you want to know what depth routing looks like from the inside, where a small model keeps its register machinery, or how a preview checkpoint has organized itself partway through training, this is the dataset.

This is a census-and-contrast run. It carries no attention-circuit spectra and no surgery pass, and the coverage section below says exactly which tables are empty.

What was run

• —Model: Smilyai-labs/Nova-1-Standard-1.3B-Preview
• —Corpus: 8,965 prompts across the standard 17 general-purpose buckets
• —Layers probed: all 24
• —Contrast: authentic against corporate register, recorded in compliance_behaviour_features
• —Passes: activation census, feature taxonomy, per-head analysis, coactivation, code-analysis, register contrast, logit lens on the attention component
• —Not run: OV-circuit SVD, Sub-Zero surgery, capability fence, SAE features

Architecture notes

This is a preview checkpoint, and that shapes how the whole atlas should be read. The model has seen roughly 4.0B pretraining tokens against 1.27B parameters, about 3 tokens per parameter, where compute-optimal guidance sits nearer 20. Every number here describes a model partway through its training run rather than a finished artifact. Where a result looks thin, training budget is the first explanation to reach for, and finding 8 is the clearest case of it.

The Mixture-of-Depths configuration is the structural feature worth knowing before reading the tables. Every second layer runs at 50% capacity, meaning only half the tokens are routed through that block while the rest bypass it. Because the census aggregates per prompt rather than per token, this atlas cannot measure routing capacity directly, but the parity signature it leaves behind is visible and is covered in finding 1.

The features row count closes exactly against this geometry at 592,896, which is 24 layers of 5,504-wide mlp, gate, and up, plus 2,048-wide attn, heads, and q, plus 1,024-wide k and v.

What the tables contain

Key findings

1. The Mixture-of-Depths parity shows up in the gate path

The architecture routes only half the tokens through every second layer. The census cannot see token routing directly, but it leaves a parity trace.

Comparing each even layer against the odd layer immediately after it, the even layer has the lower mean activation rate in 11 of 12 pairs. Aggregated:

The effect is concentrated almost entirely in one component:

Every component except gate sits within 0.01 and half of them flip sign. The gate gap is roughly nine times larger than the next largest and points the same way as the config.

The split the census recovers matches the documented architecture exactly. The model is specified as 24 layers built from 12 full blocks and 12 Mixture-of-Depths blocks, and the parity trace lands on a clean 12 and 12.

The gate is the natural place for this to surface, since it is the component whose job is deciding whether a coordinate contributes at all. On capacity-limited layers it has fewer tokens to fire on and the per-prompt activation rate drops accordingly. Read this as a signature consistent with depth routing rather than a measurement of it. The honest version needs per-token capture, which this run did not do.

2. Register separation is carried by attention, not by the MLP

Mean F-stat on the authentic-versus-corporate contrast, by component:

Query projections separate the two registers at roughly 1.8× the rate of the MLP output, and the whole attention family outranks gate and mlp.

The per-head table says the same thing more sharply. Best-head F-stats average 399.8 on q and 367.8 on k, against 330.2 for heads and 326.3 for v. The single strongest register direction in the model is layer 5 q feature 826 at 1,161, separating means of -1.177 on the authentic side and -3.611 on the corporate side.

The MLP holds most of the parameters here and the attention machinery holds most of the register distinction.

3. Register separation rises to layer 5 and then holds flat

Layer 0 is very low at 15.4, separation climbs steeply to a peak of 56.2 at layer 5, then settles into a flat band between roughly 41 and 47 for the remaining eighteen layers.

That plateau is the notable part. There is no decay toward the output. Once the model has established the register distinction by layer 5, it carries it at close to constant strength all the way through, rather than resolving it early and moving on. Seven of the ten strongest register features sit in layers 4 through 6, so the sharpest individual directions are still concentrated near the front even though the average holds up.

4. The gate is the one component with a register lean

Across every component the authentic and corporate sides split near evenly. The exception is gate:

Every other component sits within a few hundred coordinates of even. The gate runs about 24,000 coordinates heavier on the authentic side, a 59/41 split.

Read that against its baseline before drawing conclusions. The gate is the one component with a strongly positive resting activation at +0.155, where every other component sits within 0.013 of zero, and it leaves 38.2% of its coordinates idle. The sign of a delta on a positively-biased, partly-idle component is not the same measurement it is on a component centered near zero. The skew is worth recording and worth being careful with.

5. The network gets busier and denser with depth

After a dip at layer 2, activation rate climbs steadily by about 26% from there to the output, and dead coordinates fall by nine points across the same span.

Discriminative power does not follow. Mean census F-stat is essentially flat across the whole network, running between 20.5 and 25.4 with no trend. The back half is using more of its coordinates more often without those coordinates becoming better at distinguishing prompts.

6. The gate is the only strongly self-correlated component

Gate coordinates co-fire with each other at 0.553 while every other component except heads sits within 0.02 of zero.

That fits the gate's role. It is the component deciding which coordinates contribute, so it moves in blocks as whole regions of the MLP switch on together for a prompt. The up and mlp coordinates those gates control show no such structure, which means the gating is correlated even though what it gates is not.

7. KV-paired heads are near-identical at matched coordinates

Query heads pair up two to a KV head. In the heads component a feature index is head*128 + d, so an offset of exactly 128 is the same within-head dimension in the partner head.

At 0.95, two heads sharing a KV projection are running very close to the same computation at matched coordinates. With only two query heads per group, that means half the query heads in this model are close to redundant with their partner.

The same-group figure across the whole table is much lower at 0.144, because most same-group pairs are two coordinates inside a single head rather than the same coordinate across the pair. The offset-128 number is the one that isolates partner-head redundancy.

8. The output vocabulary has not settled, which is what a 3-tokens-per-parameter checkpoint looks like

The logit-lens pass reaches a mean F-stat between 48.9 and 80.1 by layer, with a single best feature at 122.2. Signal is flat across depth with no peak worth naming.

The token lists match that picture. Layer 1 attn 74, the strongest feature in the pass, promotes Chase, etooth, conn, 770, Blink, Croatian, infect, which has no coherent theme. Layer 23 attn 1891 promotes crews, Connect, prank, Carpenter, compliment, timeframe. Most other high scorers look similar.

This is the expected result at this training stage rather than a property of the architecture. The logit lens reads a direction by projecting it through the unembedding, so it can only recover coherent token families once the mapping from residual directions onto output vocabulary has converged. At roughly 3 tokens per parameter, that mapping is still forming.

The taxonomy tells the same story from another angle. Zero domain-specific directions resolved anywhere in the model, with all_shared the largest class at 32.7% and non_activated at 30.9%. Representations that have not yet differentiated.

The practical consequence is narrow. Do not mine logit_lens here for steering targets. The census, contrast, and coactivation tables are measuring structure that has formed, and findings 1 through 7 rest on those.

What the register contrast is measuring

The contrast pass looks for directions that separate an authentic register from a corporate one, scoring every feature by how reliably it distinguishes the two prompt sets. It is a correlational measurement. Nothing in this run intervenes on the model, so a high F-stat means a direction is a good classifier of the two conditions, not that removing or steering it would change behavior. The surgery passes that would test the difference were not run.

Important caveats

• —This is a mid-training checkpoint, and that is the first explanation for anything thin. Phase 3 SFT at step 5,707, roughly 4.0B pretraining tokens against 1.27B parameters. Structure that has not formed yet will read as absent, and structure still forming will read as noisy. Finding 8 is the clearest instance. None of it generalizes to what this architecture looks like trained to a compute-optimal budget.
• —No surgery pass ran, so nothing here speaks to editability. subzero_layer, subzero_svs, and subzero_capability are all empty. There are no causal scores, no DAS axes, and no capability fence. Every finding in this card is correlational.
• —No OV-circuit pass ran. ov_circuits is empty, so there are no attention spectra, no effective-rank figures, and no induction scores. Finding 7 rests on the coactivation table instead.
• —The logit lens covers the attention component only. All 1,536 rows are attn. There is no MLP-family logit-lens coverage, so finding 8 describes attention directions and says nothing about what gate, up, or mlp promote.
• —The run directory was named for a code run, but the contrast scored here is register. The corpus is the standard 17-bucket general-purpose set at 8,965 prompts, and the populated contrast table is compliance_behaviour_features, authentic against corporate. Nothing in this atlas measures code against non-code.
• —The Mixture-of-Depths result is a signature, not a measurement. The census aggregates per prompt and cannot observe token-level routing. Finding 1 reports a parity pattern consistent with the documented configuration.
• —`coactivation` stores a selected subset of feature pairs, not a full census. The comparisons in findings 6 and 7 are relative differences within that subset, meaningful as contrasts but not population means.
• —No domain-specific directions resolved, and per_head.n_specific is zero everywhere. Two causes are mixed together: general-purpose corpus buckets a model has little reason to build detectors for, and a training budget at which specialized directions would not have differentiated yet. This atlas cannot separate them.
• —Coactivation buckets describe the prompt mix. Dominant buckets come out roleplay (11.4%), design (9.6%), and business (9.4%), with 17.2% uncategorized.
• —The paired-head result is correlational. A 0.95 correlation is a strong merging signal, not proof that removal is free.
• —No downstream benchmark is implied. The atlas describes what the tensors do on this corpus, not whether the model is good at your task.

How to use

atlas.sqlite is the primary query surface. PRAGMA integrity_check returns ok, and the features row count closes exactly against the model geometry at 24 layers × 24,704 coordinates.

import sqlite3
import pandas as pd

conn = sqlite3.connect("atlas.sqlite")

# which components actually carry the register distinction?
df = pd.read_sql_query("""
    SELECT component,
           ROUND(AVG(fstat), 1) AS mean_fstat,
           ROUND(MAX(fstat), 1) AS max_fstat
    FROM compliance_behaviour_features
    GROUP BY component
    ORDER BY mean_fstat DESC
""", conn)

The per-layer JSON under layers/ and the pooled summaries under cross_layer/ mirror the same data if you would rather not open the database.

-- the depth-routing parity trace, one component at a time
SELECT component,
       ROUND(AVG(CASE WHEN layer_id % 2 = 0 THEN activation_rate END), 4) AS even_layers,
       ROUND(AVG(CASE WHEN layer_id % 2 = 1 THEN activation_rate END), 4) AS odd_layers
FROM features
GROUP BY component
ORDER BY (odd_layers - even_layers) DESC;

License

Apache 2.0, matching the source model. Consult the source model repository before redistribution or downstream use.

Contact / more

• —Model: https://huggingface.co/Smilyai-labs/Nova-1-Standard-1.3B-Preview
• —Atlas code: https://github.com/JuiceB0xC0de/qwip_atlas
• —Follow: https://huggingface.co/juiceb0xc0de