juiceb0xc0de/Qwythos-9B-Claude-Mythos-5-1M-atlas

juiceb0xc0de/Qwythos-9B-Claude-Mythos-5-1M-atlas

A brain atlas for [empero-ai/Qwythos-9B-Claude-Mythos-5-1M](https://huggingface.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M), a 32-layer hybrid that runs linear attention on 24 layers and full attention on the other 8. This is not a chat dataset or a benchmark. It is an internal-mechanics map built by running activations through a corpus of prompts and scoring what each layer, component, head, and feature direction is doing.

The interesting thing about this model is how little of it is full attention. Three layers out of every four mix sequence information through a gated linear path, and only the fourth gets keys, queries, and values in the usual sense. If you want to know what those eight attention layers end up doing when they are the only ones doing it, where a model like this keeps its output vocabulary, or which parts of it survive surgery, this is the dataset.

What was run

• —Model: empero-ai/Qwythos-9B-Claude-Mythos-5-1M
• —Corpus: 8,965 diverse prompts across 17 buckets
• —Layers probed: all 32
• —Full-attention layers: 3, 7, 11, 15, 19, 23, 27, 31, with per-head capture (attn, heads, q, k, v)
• —Linear-attention layers: the other 24, captured as linattn_qkv, linattn_z, and linattn_out
• —MLP components on every layer: mlp, gate, up
• —Contrast: authentic against corporate register
• —Passes: activation census, feature taxonomy, per-head analysis, OV-circuit SVD, logit lens, coactivation, code-analysis, register contrast, Sub-Zero surgery with capability fence across code, math, reasoning, factual, and multilingual

Architecture notes

Three structural facts shape how the tables should be read.

The layer types are not interchangeable, and the census reflects that. A linear-attention layer carries six components and 53,248 coordinates; a full-attention layer carries eight and 55,296. The features row count closes exactly against that split at 24 × 53,248 + 8 × 55,296 = 1,720,320, and head_idx // 4 equals kv_head on all 128 rows of ov_circuits.

The q component is 8,192 wide where 16 heads at 256 dimensions would give 4,096. That is the attention output gate: q_proj emits the query and its gate concatenated, so half those coordinates are gating signal rather than query. The per-head pass split the width by head dimension and produced 32 rows per layer for a 16-head model. The upper 16 are gate coordinates. This matters for reading per_head and is covered again in the caveats.

The linear-attention components pack differently from attention. linattn_qkv is 8,192 wide, consistent with 16 key heads and 16 query heads at 128 dimensions plus 32 value heads at 128; linattn_z at 4,096 is the value-path gate; linattn_out at 4,096 is the projection back into the residual stream. These are the projections around the recurrence, not the recurrence itself.

What the tables contain

Key findings

1. The model is bimodal: half of it is nearly silent, a third is nearly always on

Across all 1,720,320 coordinates, activation rate does not spread out. It piles up at both ends.

Half the coordinates in this model fire on one prompt in twenty or fewer. Another third fire on nine prompts in ten. Only about one coordinate in six sits in the context-dependent middle where a feature can be selective about what it responds to.

The taxonomy is the same fact under different labels: non_activated at 49.96%, all_shared at 34.43%, partial_shared at 9.06%, broadly_shared at 6.54%.

That shape is the thing to hold onto while reading the rest of this card. Most findings below are about what that middle sixth is doing, because it is where the discriminative work happens, and about which components own it.

2. The gate is the sparse component, and it is where this model keeps its meaning

Census statistics by component, sorted by how much of each sits idle:

The three gating components sit at the top and everything else clusters near a 0.50 rate and a resting activation of zero. gate, q, and linattn_z are the only components with a strongly negative resting bias, which is what a pre-sigmoid gate looks like from the census side.

The gate is not just sparse. It is the most selective component under code-analysis at 84.9% selective against 63.8% for k, it holds 54 of the 77 domain-specific directions the taxonomy resolved anywhere in the model, and it is the strongest logit-lens component at a mean F-stat of 289.8, running 1.89× the mlp path and 2.38× linattn_out.

Sparse, negatively biased, selective, specialized, and legible at the output are one behavior described five ways. The component that decides what contributes is also the component whose contributions are easiest to read.

3. Two query heads in every full-attention layer do almost no induction

Induction across the 128 head-circuits averages 0.856, and 109 of 128 heads score above 0.8. Nineteen do not, and they are not scattered.

Every one of the 16 head-circuits scoring below 0.5 belongs to kv_head 0. Inside that group the split is by head index:

Heads 0 and 2 are near the floor in all eight attention layers. Heads 1 and 3, sharing the same keys and values, are near the top. The two populations do not overlap at any layer.

The alternation is not confined to group 0. It runs model-wide, just much more gently:

Odd-indexed heads average 1.002 and even-indexed heads 0.709 across the whole table. In groups 1 through 3 that gap runs 0.10 to 0.13. In group 0 it is 0.82.

Two things are worth separating here. The gentle even/odd alternation is present everywhere and is small enough to be a property of how the metric interacts with head ordering. The collapse in group 0 is a different size of effect entirely, it reproduces in all eight layers independently, and it says that the first KV group is running two heads that do something other than copy-and-continue while its other two heads do little else. What that something is would need an ablation to answer, and this atlas cannot.

4. Induction is spread evenly across all eight attention layers rather than concentrated

The spread from the highest layer to the lowest is 0.060. There is no induction layer and no induction band; every full-attention layer carries roughly the same amount.

Read that against the architecture. Only one layer in four gets full attention at all, and each of those eight is doing comparable copy-and-continue work rather than the job being handed to a specialist pair of layers. The 24 linear-attention layers in between are not visible to this measurement, so this describes how the attention budget is used, not how sequence mixing is divided across the model.

Note that induction_score exceeds 1.0 on 31 head-circuits and tops out at 1.121, so it is not normalized to a unit interval and should be read as a relative score.

5. Attention routing narrows sharply in the last two attention layers

QK effective rank halves and spectral concentration doubles at layers 27 and 31. The OV path narrows too, but far less, from 99.6 to 78.0.

Across the whole table the OV path runs at 36.8% of the 256-dimensional head and QK at 15.5%. The transform stays broad while the routing decision stays narrow, and in the last two attention layers the routing gets narrower still while induction holds steady at 0.846 and 0.861.

Effective rank does not transfer across architectures without normalizing by head dimension, so treat these as fractions rather than raw numbers.

6. The logit lens is legible, gate-driven, and carries concepts across languages

Signal builds steadily with depth:

By component, gate leads at 289.8, then up at 216.5, heads at 187.8, mlp at 153.6, attn at 125.2, and linattn_out at 121.9.

The top features carry coherent token families, and several of them are coherent across scripts rather than across spellings:

• —Layer 23 gate 5067 promotes concepts, 概念, concept, Concepts, 概念的, 的概念, Concept
• —Layer 28 gate 4316 promotes 故事, stories, Stories, 故事中, story, 这个故事
• —Layer 24 gate 9977 promotes 统计, statistical, statistics, 统计分析, 統計, statistic
• —Layer 20 gate 5655 promotes Layers, layers, keras, Dropout, softmax, KER
• —Layer 31 gate 9995, the strongest in the pass at 630.8, promotes soundtrack, directed, жан, Trailer, 豆瓣, indie, sequel, genre

The statistics feature ties simplified and traditional Chinese to the English word in one direction. The film feature reaches a Chinese film-database term and a Russian fragment of "genre" alongside the English. These are concept-level directions rather than token-level ones, and they are the usable steering targets in this pass.

Not every high scorer is like this. Layer 18 gate 2221 scores 600.7 while promoting trys, onte, 架, ovac, 意识的, OperationException, Greetings with no shared theme. Check the token list before treating any single direction as a target.

7. Register separation is MLP-led and peaks in the middle of the stack

Mean F-stat on the authentic-versus-corporate contrast, by component:

The mlp output separates the two registers at 1.52× the rate of the heads component. Of the fifty strongest individual register directions, 21 are mlp, 14 are gate, and 9 are up, so 44 of 50 sit in the MLP family.

By depth the axis builds and then decays:

Layer 0 is very low at 7.9. Separation climbs by a factor of 8.3 to a peak of 65.6 at layer 15, then falls to 33.7 at the output. The single strongest direction is layer 3 heads 3761 at 1,270.8, which is the one place the attention family tops the table.

Only two components carry a directional lean. gate splits 182,965 authentic-leaning against 210,247 corporate-leaning, a 47/53 skew, and q splits 35,263 against 30,271 the other way, 54/46. Every other component sits within a few hundred coordinates of even. Both leaning components are gates with strongly negative resting activation, so the sign of a delta there is not the same measurement it is on a component centered at zero.

8. The register direction is 95% decodable at every layer, however concentrated it is

subzero_layer fits a linear classifier for the register contrast at each of the 32 layers:

Accuracy is flat. Layer 0, where per-feature F-stat is 7.9, decodes register at 0.953. Layer 15, where per-feature F-stat is 65.6, decodes it at 0.953 as well.

Those two measurements are not in conflict, and the gap between them is the useful part. A linear probe over the full residual width can find the register distinction anywhere in the network. What changes with depth is how much of that distinction has collected into individual coordinates. Finding 7 describes concentration, not availability.

The practical consequence is that if you want a direction you can name and edit, depth matters and layers 12 through 17 are where to look. If you only want to detect register, any layer will do.

9. The linear-attention gate trades activity for discrimination with depth

linattn_z runs in opposite directions on two measurements:

The gate opens through the first half, peaking at 0.787 at layer 17, then closes hard, bottoming at 0.119 at layer 26. Over the same span its census F-stat rises by a factor of 2.4.

Late linear-attention layers gate a quarter as often and are more than twice as discriminative when they do. This is the same trade the gate component makes on the MLP side, arriving through depth rather than by component.

The whole-model census moves with it. Quiet coordinates hold near 48% through layers 0 to 23 and rise to 53.3% in layers 24 to 31, while mean census F-stat peaks at layers 24 and 25 at 60.7 and 60.5.

10. The four axes that fail the capability fence are all in the last two layers

The surgery pass retained 12 singular vectors and tested each against five capability domains, 60 rows. Eight axes pass all five domains and four fail all five.

The two populations do not overlap. Passing rows run 0.0005 to 0.0583 nats per token and failing rows run 0.1434 to 0.3847, a mean of 0.0305 against 0.2703, a factor of 8.9 with a clean gap between the worst pass and the best failure.

Every failure is a gate_proj or up_proj axis at layer 30 or 31. All ten down_proj rows and all ten linattn_in_proj_z rows pass, including down_proj at layer 31 sitting in the same layer as two of the failures. Damage is also near-uniform across domains, with means from 0.100 on math to 0.116 on factual, so no single capability is carrying the failures.

The reading is that in this model the entry projections of the final two MLP blocks are load-bearing for general capability and the exit projection of the same block is not. That is a small, specific result on 12 axes, and the next section says why it should not be read as a map of the model's editability.

What Sub-Zero is measuring

The Sub-Zero pass is not a generic "find all important directions" sweep. It looks for directions that separate corporate style from authentic style, then uses DAS rotation and a capability fence to check whether removing those directions damages code, math, reasoning, factual, or multilingual ability. The rows in subzero_capability are domain-by-domain damage scores for those candidate axes, not a census of every load-bearing direction in the model.

On this run the candidate stage was highly selective. compliance_behaviour_sv is zero on 24 of the 32 layers, meaning most layers contributed no retained direction at all, and the 12 that survived came from eight layers. Finding 10 describes those 12 axes and nothing else.

Important caveats

• —The surgery pass is thin, and finding 10 does not describe this model's editability. Twelve axes across eight layers, against 24,576 candidate singular values per linear-attention layer. Four projection families were reached (gate_proj, up_proj, down_proj, linattn_in_proj_z) and the attention projections were not reached at all. The clean depth split is a real result about 12 directions and not a survey of where this model can be edited.
• —The SSM recurrence is not probed. linattn_qkv, linattn_z, and linattn_out are the projections around the gated linear-attention path, not the recurrent state, the convolution, or the decay terms. On a model where 24 of 32 layers mix sequence information through that path, a large share of the machinery is outside what this atlas can see, and every finding here describes the MLP, attention, and linear-attention projection surfaces only.
• —`per_head` reports 32 `q` heads for a 16-head model. q_proj is 8,192 wide because the attention output gate concatenates the query and its gate, and the per-head pass split that width by head dimension. The upper 16 rows per layer are gate coordinates, not query heads. Both halves behave near-identically in the census, at an activation rate of 0.2745 each and quiet shares of 68.95% and 68.99%, so nothing in the aggregate q statistics is distorted; the per-head row count is.
• —`ov_circuits.compliance_score` and `ov_circuits.layer_comp_strength` are NULL on all 128 rows. They carry no information for this run. subzero_layer.sv_total is a constant 24,576 on linear-attention layers and 12,288 on full-attention layers, so it encodes layer type and nothing more.
• —`induction_score` is not bounded at 1. It exceeds 1.0 on 31 of 128 head-circuits and peaks at 1.121. Treat it as a relative ranking within this table.
• —The head-redundancy comparison in the coactivation table is confounded. All 1,020 matched-coordinate head pairs in the stored subset fall inside a GQA group, and nearly all cross-group pairs are unmatched-coordinate. Within-group matched pairs correlate at 0.899 and cross-group pairs at 0.269, but that contrast mixes the group boundary with the coordinate match and cannot separate them. It is a merging signal, not proof that removal is free, and it would need a fenced ablation run.
• —`coactivation` stores a selected subset of feature pairs, not a full census. The component comparisons are relative differences within that subset, meaningful as contrasts but not population means.
• —Only 77 domain-specific directions resolved out of 1,720,320 features, 54 of them in gate, spread across layers 3 to 31. The corpus buckets are general-purpose categories. Before reading that as a statement about how specialized this model is, read it as a statement about what a general-purpose corpus can see. Answering the specialization question needs a corpus split along axes the model was actually post-trained on, and that is a follow-up run rather than a conclusion.
• —Coactivation buckets describe the prompt mix, and this one is lopsided. Dominant buckets come out business at 30.8% and introspection at 4.2%, with 43.0% carrying no bucket at all. Those proportions reflect the corpus and the pair-selection step, not the model.
• —One behavioral axis only. This run scored the authentic-versus-corporate register contrast. Nothing here speaks to content domain, refusal, or reasoning depth.
• —The vision tower and the MTP head are absent. The config carries a 27-layer, 1,152-wide vision encoder and one multi-token-prediction layer. The text-only load path drops both, so neither appears anywhere in this atlas.
• —The census aggregates per prompt, not per token. Activation rate is the fraction of the 8,965 prompts on which a coordinate fires, which is why the bimodal shape in finding 1 is a statement about prompt-level consistency rather than token-level sparsity.
• —Effective rank is not comparable across model families without normalizing by head dimension. This model's heads are 256-dimensional.
• —Damage is in nats per token, measured on the fence probes, not on any public benchmark.
• —No SAE features. The sae_features table exists but is empty for this run.
• —No downstream benchmark is implied. The atlas describes what the tensors do on this corpus, not whether the model is good at your task.

How to use

atlas.sqlite is the primary query surface. PRAGMA integrity_check returns ok, and the features row count closes exactly against the model geometry at 24 linear-attention layers × 53,248 coordinates plus 8 full-attention layers × 55,296.

import sqlite3
import pandas as pd

conn = sqlite3.connect("atlas.sqlite")

# half quiet, a third saturated: where does each component actually sit?
df = pd.read_sql_query("""
    SELECT component,
           ROUND(AVG(activation_rate), 4)                     AS mean_rate,
           ROUND(100.0*AVG(activation_rate <= 0.05), 2)       AS pct_quiet,
           ROUND(100.0*AVG(activation_rate >= 0.90), 2)       AS pct_saturated
    FROM features
    GROUP BY component
    ORDER BY pct_quiet DESC
""", conn)

The per-layer JSON under layers/ and the pooled summaries under cross_layer/ mirror the same data if you would rather not open the database.

-- the two heads in every attention layer that skipped induction class
SELECT head_idx,
       kv_head,
       COUNT(*)                       AS layers,
       ROUND(AVG(induction_score), 3) AS mean_induction
FROM ov_circuits
GROUP BY head_idx
ORDER BY mean_induction
LIMIT 6;

Source model license

Apache 2.0, matching the source model. Consult the source model repository before redistribution or downstream use.

Contact / more

• —Model: https://huggingface.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M
• —Atlas code: https://github.com/JuiceB0xC0de/qwip_atlas
• —Follow: https://huggingface.co/juiceb0xc0de