ceselder/thinkies-v3

thinkies v3 — 1,583,873 concept vectors from Qwen3.6-27B

One row per short natural-language phrase, paired with the layer-42 residual-stream direction that phrase induces in Qwen3.6-27B. 2.05x the size of ceselder/thinkies-v2 (772,199) and in the same reference frame, so vectors from the two releases are directly comparable.

How each vector was made

For a phrase X, X is substituted into 16 prompt templates of the form "Focus on the following idea: X while writing the following phrase: Y", where Y is a fixed neutral carrier. Y's tokens are force-appended and the layer-42 residual stream is averaged over exactly those carrier positions. The vector is the mean of that over the 16 templates, minus a global reference mean (thinkies_v3_ref_mean.npy, shipped here).

Vectors are directions. Scale is discarded downstream (cosine, NNOLS, norm-matched injection), which is why fp16 is enough and why the reference mean must be subtracted before comparing anything: raw activations here are ~93% a shared constant, so two arbitrary raw vectors sit at cosine 0.996. Add ref_mean back if you want the uncentered activation.

reliability — what it does and does not tell you

Split-half cosine: the 16 draws are split into disjoint halves, each half mean-pooled, both contrasted against the per-template mean, then cosine between them. It measures whether the templates agree on a direction — test-retest precision. It is not a measure of whether the vector means its label, and it does not predict how uniquely the label pins the vector down.

Filters applied: reliability >= 0.65, labels of at most 8 tokens, then activation-space dedup at cosine > 0.95 via LSH bucketing, which removed 27,366 rows (1.70%) — differently-worded phrases the model represents identically.

Label length

1w 3,428 · 2w 128,475 · 3w 261,844 · 4w 338,270 · 5w 342,748 · 6w 331,692 · 7w 112,859 · 8w 64,557

Caveats

• —Labels are mined from web text and are not filtered for content.
• —Directions are specific to Qwen3.6-27B at layer 42 and to the template/carrier construction above; they are not expected to transfer to other models or layers as-is.
• —The nearest neighbours of a vector are often a semantic schema rather than the exact instance: the neighbourhood of euthanizing a puppy is dominated by violence-against-the- defenceless phrases across many victim types. Treat "closest label" accordingly.

Multi-token OMP readouts (omp_readouts-*.parquet)

Every one of the 1,583,873 vectors also has a multi-token readout: a set of 16 English tokens chosen to explain the vector, rather than the tokens with the largest logit.

How, and why not just use a J-lens

The atom is mapped to the last layer with the fitted Jacobian from camilablank/workspace-lenses (J[42], target layer 62) and then decomposed by orthogonal matching pursuit over the unembedding rows, restricted to English wordlike tokens with non-negative coefficients.

The standard single-token J-lens readout is redundant by construction — its top tokens are near-duplicates (' cost' ' costs' 'cost' 'Cost' is one direction shown four times), so it spans a tiny subspace. Measured on a held-out document, probability-weighted top-K explains cosine 0.202 at K=64 and saturates there even at K=1024, while this constrained OMP readout reaches 0.358 — 1.77x more of the vector, while staying readable.

Two caveats. Unconstrained OMP scores higher still (0.505) but does it by selecting rare undertrained tokens with enormous negative weights, which is numerically efficient and semantically worthless — hence the constraints here. And the atom is transformed centered: the shared mean has norm 65.5 against a 4.3 concept residual, so transforming the uncentered activation yields the same generic readout for every atom (verified).

Examples: Gastric lymphoma -> medical disease health biopsy diagnosis cancer clinical · Earth tone -> colors rustic brown aesthetic painted sunlight hues · Cycle stealing -> CPU security computational exploiting technology stealth machine