ksopyla/gemma3-concepts-1b-c128-1bt

gemma3-concepts-1b-c128-1bt

Gemma-3-1B with a grafted shared depth-recurrent concept memory (C=128). Research release from MrCogito experiment E16b: trained for ~1B tokens at seq 4096 on a long-document mix.

Claim level: mechanism success — concepts are causally used (ablating them raises long-range CE a lot). Generation quality is not the claim: this is a research base LM, not instruction-tuned, and open-ended text is often weak or repetitive. Do not treat samples as evidence of a strong generative model. Also not a GLUE / chat-arena SOTA claim.

Table of contents

1. 1.Model summary
2. 2.How to use
3. 3.Uses
4. 4.Limitations
5. 5.Architecture
6. 6.Training
7. 7.Evaluation
8. 8.Citation
9. 9.License

Model summary

What it is

• —A block-recurrent causal LM: tokens run through Gemma in K=512 blocks; after each global layer the model reads/writes a shared concept state z ∈ R^{C×H}.
• —Evidence that those concepts become necessary for next-token loss at long positions (ablating them raises CE a lot).

What it is not

• —Not a strong generator. Continuations can be dull, repetitive, or locally fluent without being useful. The published success is concept use under ablation, not sample quality.
• —Not chat / instruction-tuned — use continuation prompts, not “system” roles.
• —Not loadable as plain AutoModelForCausalLM without MrCogito.
• —Not evaluated here on STS-B / GLUE / arena — those probes are listed as not run.

How to use

git clone https://github.com/ksopyla/MrCogito.git
cd MrCogito
uv sync

Accept the Gemma license on Hugging Face if prompted.

import torch
from transformers import AutoTokenizer
from nn.backbone_concept_lm import BackboneConceptLM

repo_id = "ksopyla/gemma3-concepts-1b-c128-1bt"

if torch.cuda.is_available():
    device, dtype = "cuda", torch.bfloat16
elif getattr(torch.backends, "mps", None) and torch.backends.mps.is_available():
    device, dtype = "mps", torch.float32   # fp16 often NaNs on Apple MPS
else:
    device, dtype = "cpu", torch.float32

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = BackboneConceptLM.from_pretrained(repo_id, torch_dtype=dtype).to(device).eval()

prompt = "Once upon a time, in a quiet library at the edge of the city,"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )
print(tokenizer.decode(out[0], skip_special_tokens=True))

Research API — concept ablation (same generate / forward path):

# concept_mode: "real" | "zero" | "shuffle" | "static" | "one_block"
out = model.generate(**inputs, max_new_tokens=32, concept_mode="shuffle")

Playground notebook: playground/e16b_generation_playground.ipynb.

Uses

Direct use

• —Reproduce E16b Tier-1 concept geometry and causal-ablation numbers.
• —Ablation studies showing concepts are used (concept_mode=…).
• —Starting checkpoint for longer-context or semantic-probe follow-ups.

Out of scope

• —Showcasing high-quality generation or shipping a writing/chat assistant.
• —Production assistants, customer-facing chat, or safety-critical decisions.
• —Claiming parity with Gemma IT / chat models.
• —Multilingual deployment (training mix is English-centric).

Limitations

1. 1.Generation is restricted / weak for product use. Prefer this checkpoint for concept ablations, geometry, and mechanism studies — not demos of “good writing.” Formal generation vibe-check metrics are not published; qualitative play only.
2. 2.Custom code — BackboneConceptLM lives in MrCogito; clone the repo.
3. 3.Base LM, not IT — will not “follow instructions” like a chat model.
4. 4.Slow generate — no KV cache in the shipped loop; keep max_new_tokens small (≈64–128 on laptop GPU/MPS).
5. 5.Apple MPS — use float32.
6. 6.Compound bet — long context + long-doc mix + Muon + 1B tokens were changed together; factor isolation is open.
7. 7.Gemma license applies to derived weights.

Architecture

Vision (why concepts at all)

MrCogito is a research bet that reasoning should eventually happen in a compact concept space, not only as next-token prediction over text:

long input (text today; audio/vision later)
    → compress into C latent concept vectors   (C ≪ N)
    → refine / reason in concept space         (recursion, depth, …)
    → decode / crystallise back to tokens      (training signal + human interface)

The belief behind that stack:

1. 1.Semantic bandwidth. A text token carries on the order of ~15 bits; a concept vector in a large hidden space carries far more continuous state per step. That gap is the argument for latent reasoning and (later) latent agent-to-agent channels.
2. 2.Long context as a consequence of the bottleneck. If attention is concept↔token (roughly O(C·N) with C ≪ N) rather than full O(N²) self-attention, million-token windows become a design goal of the architecture, not a RoPE trick bolted onto a dense transformer.
3. 3.Decode is crystallisation, not the thinking substrate. Tokens are how we train and how humans read answers; the interesting state should live in concepts.
4. 4.One step at a time. Multimodality and multi-agent concept exchange are enabled by a working concept core — they are not Stage-0 goals. Near-term priority is proving concepts actually carry usable state.

Full write-up: vision and goals · living agenda · ai.ksopyla.com.

Where this release sits

E16b is not the full encode→recursive-reason→decode stack yet. It is a Gemma graft: keep a strong frozen decoder LM, add a small shared concept workspace, and ask a sharper mechanism question —

when next-token loss can already be solved from local Gemma context, do concepts ever become causally necessary?

Earlier short-context runs (E10–E16a, seq 2048, ≤100M tokens) often kept concept geometry healthy (high RankMe) while beyond-local ablation ΔCE stayed near zero. E16b changes the operating regime (seq 4096, long-document mix, Muon, 1B tokens) and clears that gate: ablating concepts at long positions hurts a lot. That validates one path for causally load-bearing concept memory — a platform to build recursive / latent-reasoning ideas on — without closing other routes (from-scratch AR, diffusion, etc.).

This model’s forward pass

input_ids [B, L]
    │
    ▼  consume in blocks of K=512
┌───────────────────────────────────────────────┐
│  Gemma-3-1B decoder (frozen) + LoRA adapters  │
│                                               │
│  local / sliding layers  →  standard Gemma    │
│  global (full-attn) layers:                   │
│      1. run original Gemma layer              │
│      2. READ  concepts → tokens (gated xattn) │
│      3. WRITE tokens → concepts (BiXT update) │
└───────────────────────────────────────────────┘
    │
    ▼
shared concept state  z [B, C=128, H=1152]
    │  carried across blocks (recurrent memory)
    ▼
LM head @ last position → next-token logits

Mechanics in short:

• —Block recurrence. Sequence length 4096 ⇒ 8 blocks of K=512. Concepts are the only compact state that must survive from block to block.
• —Shared depth-recurrent I/O. The same write head is applied at each of Gemma’s global layers (depth-tied), with small tanh gates (init 0.01) and RMSNorm on the concept side of the read — so the workspace can accumulate multi-block content without free-riding on local Gemma context alone.
• —Read. After a global layer, tokens attend to the concept set (no RoPE on the concept side — the memory is a position-free set).
• —Write. BiXT-style bidirectional update mixes the current block’s tokens into z, gated so early training can stay near the frozen backbone.
• —Ablations. zero / shuffle / static / one_block break the memory on purpose; large ΔCE means the intact model was using it.

Code: `nn/backbone_concept_lm.py`.

Training

Data

Training used the frozen mix recipe `e16b_long_4k_v1` — a long-document-heavy causal-LM mix at seq length 4096, designed so enough documents span multiple concept blocks (target: ≥40% of docs >2k tokens, ≥20% >4k).

Mix proportions (sampling weights)

Weights are document-sampling mixture weights from the recipe (sum = 1.0). They are not guaranteed equal to final non-padding token mass after truncation, but they define the intended corpus balance.

Long vs fluency split (by weight): long/coherent tier ≈ FinePDFs 30% + PG19 18% + Wikipedia 12% = 60%; fluency/code/math ≈ DCLM 15% + FineWeb-Edu 10% + Stack-Edu 10% + FineMath 5% = 40%.

Design notes

• —Why this mix: short 2K fluency-heavy mixes let Gemma satisfy next-token CE from local context alone. E16b deliberately ups multi-block pressure with PDFs + books at 4K (8×512 concept blocks).
• —No cross-doc packing: each training example is one document (truncated to 4096), so concept carry across blocks is within-document, not an artifact of stitching.
• —Deferred sources (not in this mix): peS2o, Nemotron, ProLong (format / gated / MDS constraints at the time of the recipe).
• —Wikipedia license: CC-BY-SA — attribute when redistributing Wikipedia-derived text.
• —Eval reuse: Tier-1 concept analysis also sampled from the same pretokenized e16b_long_4k_v1 Gemma-4K manifest (see Evaluation).

Tokenizer

Load either from this repo or from the backbone id — they match the training tokenizer.

Procedure

Compute

Evaluation

Protocol (Tier-1, 2026-07-25): pretokenized e16b_long_4k_v1, seq 4096, buckets 1024,2048, 24 docs × 2 seeds.

Summary: within-sample RankMe stays high (~101/128) and beyond-local concept ablations clear the pre-registered ≥0.01 gate by a large margin (min(Δstatic, Δshuffle)_beyond ≈ 2.35 nats).

Concept geometry

Causal concept use (ΔCE, nats)

Higher Δ ⇒ ablating concepts hurts next-token CE (model was using them).

Registered gate

Same-family context (not length-matched)

Not measured / not claimed

Citation

@misc{sopyla2026gemma3-concepts-1b-c128-1bt,
  author       = {Sopyła, Krzysztof},
  title        = {gemma3-concepts-1b-c128-1bt (E16b)},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/ksopyla/gemma3-concepts-1b-c128-1bt},
  note         = {Gemma-3-1B + C=128 shared-depth concepts; 1B-token long-ctx Muon}
}

Please also respect the Gemma 3 terms / base model card.

License

license: gemma — weights derived from `google/gemma-3-1b-pt`.

Model card contact

Krzysztof Sopyła — https://ai.ksopyla.com · https://github.com/ksopyla