cognica/Cognica-PoE-v1.0-3B-base

Cognica-PoE-v1.0-3B-base

A 3.02B parameter causal language model pretrained from scratch with Product of Experts (PoE) per-stage-head local learning. The model has 4 PoE stages with asymmetric layer counts (16, 6, 5, 5) — stage 0 (16 layers, ~50% of trunk) acts as a high-capacity general-LM backbone, while stages 1-3 (6+5+5 deeper layers) refine specialty knowledge. Each PoE stage has its own additive lm_head that composes with the shared base lm_head:

logits_k = lm_head(x_k) + lm_head_stages[k](x_k)    for k in 0..3

Inference aggregates per-stage log-softmax distributions (Bayesian PoE, uniform mean / alpha=0.0).

This is a mid-training research release. Final training plan is 83,923 steps (~66B tokens, Chinchilla ratio ~22). Multiple checkpoints are released as branches step-XXXXX (see "Checkpoints" below). main tracks the latest.

TL;DR

• —3.02B params: 2.08B transformer trunk + 0.54B value-embeds + 0.34B lmheadstages + 0.07B wte
• —Architecture: depth=32, nembd=2048, nhead=16, nkvhead=8 (GQA 2:1), headdim=128, intermediatesize=12800, maxseqlen=2048
• —PoE: K=4 stages, asymmetric poe_stage_layers=(16, 6, 5, 5), boundaries at layers [15, 21, 26, 31], poe_mode=flat, poe_alpha=0.0 (uniform stage mean)
• —Per-stage heads: 4 independent additive lmheadstages composing with shared lm_head
• —Training: DistMuonAdamW (ZeRO-2), totalbatch=786,432 tokens/step, ~66B target tokens, Chinchilla ratio ~22, FA2, bf16 compute / fp32 weights, `caseaug_prob=0.15`
• —Dataset: frontier_v1 mix (63B tokens), 11 sources covering English / multilingual / code / math / books / chat
• —Tokenizer: 32,768 BPE vocab, BOS-prepend protocol (see "Inference" below)
• —Standard HF AutoModelForCausalLM + AutoTokenizer with trust_remote_code=True
• —WAND p99 bounds are now per-checkpoint, stored in `config.json` (auto-calibrated; class-constant fallback only)

Architecture details

Stage layout

Stage 0 is intentionally deep enough to function as a standalone capable LM. The asymmetric layout (50% / 19% / 16% / 16%) is itself a research variable: see the "Diversity vs layout" note below.

Training

Dataset (frontier_v1 mix, 63.07B tokens, 848 sharded parquets)

Checkpoints

Each ckpt is a separate branch named step-XXXXX. The main branch tracks the latest released checkpoint (currently `step-83923` — final, training complete).

Training-log val BPB new-minimum trajectory: s24500=0.9216 → s26500=0.9205 → s27000=0.9170 → s27500=0.9152 → s29000=0.9150 → s30000=0.9139 → s30500=0.9058 → s32000=0.9029 → s33500=0.9025 → s35000=0.9019 → s35500=0.8957 → s37500=0.8936 → s40000=0.8904 → s41500=0.8856 → s42000=0.8849 → s44000=0.8827 → s44500=0.8777 → s46500=0.8735 → s47000=0.8720 → s48500=0.8686 → s49500=0.8677 → s50000=0.8655 → s50500=0.8645 → s51000=0.8604 → s51500=0.8525 → s54500=0.8542. Warmdown phase began at step 29373; LR decay (lrm) is 1.00 at start, 0.85 by step 38000, 0.74 by step 44000, 0.65 by step 50000, 0.59 by step 54000.

Load a specific checkpoint via:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "cognica/Cognica-PoE-v1.0-3B-base",
    revision="step-83923",       # branch name
    trust_remote_code=True,
    torch_dtype="bfloat16",
)
tokenizer = AutoTokenizer.from_pretrained(
    "cognica/Cognica-PoE-v1.0-3B-base",
    revision="step-83923",
    trust_remote_code=True,
)

WAND bounds (per-checkpoint, calibrated)

Each branch's config.json carries poe_wand_p99_bounds_per_stage_head, calibrated on a 131,072-token val slice using the tight margin-shrinkage metric range(delta) = max(delta) - min(delta) (constant-shift invariant). model.generate_wand(...) reads this field automatically; the class constant POE_WAND_P99_BOUNDS_PER_STAGE_HEAD = (3.2557, 1.5259, 1.1327) is now a fallback only.

The bound 0→1 decreased s2k → s18k (from peak 3.84 at s4000 to 3.09 at s18000). Subsequent windows produced repeated widening / narrowing cycles: at s20k → s28k all three rose +3-5%, descended through s30k → s36k, widened sharply at s38k (+13~14%), narrowed at s40k (-7%), split at s42k, widened uniformly at s44k (+12-5%), reverted at s46k (-5%), mild moves through s48-s56, widened uniformly at s58k (+10%; bound 1→2 = 2.0701 set a trajectory-wide single-bound high), narrowed substantially at s60k (-12-13% — the s58 widening fully reverts), held essentially flat at s62k (+0.5% / +0.1% / +3.5%), narrowed mildly through s64-s66 (-0.9% to -4.8%), widened uniformly at s68k (+13.85% / +16.74% / +18.10%), narrowed substantially at s70k (-10.34% / -11.01% / -8.56%), mildly re-widened at s72k (+1.43% / +4.56% / +2.37%), widened moderately at s74k (+8.47% / +9.50% / +6.43%), mildly narrowed at s76k (-1.33% / -3.18% / +0.15%), and split at s78k (+5.45% bound 0→1 / +0.30% bound 1→2 / -1.10% bound 2→3). The trajectory is non-monotonic on every measurement window.

Inference

Standard HF generate (with BOS prepend — REQUIRED for base ckpts)

This is a base (pretrained) model. The training protocol always prepends <|bos|> to the prompt before tokenization. Failing to prepend BOS produces incoherent output:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
    "cognica/Cognica-PoE-v1.0-3B-base",
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(
    "cognica/Cognica-PoE-v1.0-3B-base",
    trust_remote_code=True,
)

prompt = "The capital of France is"
input_ids = [tokenizer.bos_token_id] + tokenizer.encode(prompt, add_special_tokens=False)
input_ids = torch.tensor([input_ids], device=device)
out = model.generate(
    input_ids=input_ids,
    max_new_tokens=32,
    do_sample=False,           # greedy; set True + temperature for sampling
)
print(tokenizer.decode(out[0].tolist()))

KV cache is enabled by default. CognicaKVCache subclasses transformers.Cache so HF generate() preserves it across decode steps without auto-replacing it with DynamicCache. The cache is preallocated to max_position_embeddings and lives on the device of the input tensor.

Implementation details:

• —Numerical: SDPA's prefill (is_causal=True, full sequence) and decode (Tq == 1, masked) kernels are mathematically equivalent but accumulate bf16 rounding errors in different orders. To prevent that drift from compounding across decode steps and producing different greedy tokens at low-margin branching points, the SDPA call casts q/k/v to fp32, runs the kernel, then casts back to bf16. The K/V cache itself stays in bf16 (memory unchanged). On a fixed greedy prompt this gives bit-identical agreement between use_cache=True and use_cache=False for at least 200 generated tokens.
• —Throughput: in single-batch (B=1) interactive use, per-decode Python and dispatch overhead dominates the per-step compute savings from the cache. Measured speedup is +3-6 percent (use_cache=True vs use_cache=False) over 50-500 token runs. To realize the cache's full benefit, batch the decode (B >= 4) or use a fused kvcache kernel (FA2's flash_attn_with_kvcache, FlashInfer).

PoE-specific inference (s83923 final measurements, 8-shard val slice 1.05M tokens)

s83923 is the final ckpt (lrm at s83923 ≈ 0.05; warmdown complete). Same val slice across s8000..s83923, single A100 80GB, bug-fixed code:

vs s82000 the local-slice training-objective BPB dropped -0.005308 (full K=4) and -0.005332 (single s3). Sub-0.725 first crossed across the full table.

Cumulative warmdown phase totals: full K=4 BPB s30000 → s83923 = -0.133244 (15.5% relative reduction). Per-stage full acc s32000 → s83923 = +0.0580 (5.80 percentage points).

Per-stage target accuracy across last 12 ckpts (s52000 skipped from analysis):

Cumulative s32 → s82 full acc gain: +0.0555. 0.05 cumulative milestone crossed at s78k; 0.48 boundary first crossed at s80k (s2/s3/full = 0.4824 / 0.4829 / 0.4824); s82k continues +0.0011 mild descent.

Sample-level outputs at s78000 (greedy temp=0.0, 60 tokens):

• —Capital France: "Paris. It is the largest city in France and the capital of the country. Paris is the seat of the government, the seat of the French Academy, and the seat of the European Union. It is also the seat of the United Nations. Paris is the second largest city in the European Union." (Paris ✓; "largest city in France" ✓; "seat of government" ✓; "seat of French Academy" ✓ — Académie française is in Paris; "seat of European Union" wrong; "seat of UN" wrong; mixed factual quality)
• —Gold symbol: "Au. Gold is a soft, malleable, ductile, highly unreactive ✓, precious, yellow, ductile, malleable, ..." (Au ✓; "highly unreactive" ✓ — corrects the s72 "highly reactive" error; precious + yellow + soft + malleable + ductile correct; output then degenerates into ductile/yellow/malleable repetition loop)
• —Friday → tomorrow: "Saturday. If you are a Christian, then you know that the Bible says that God created the world on the sixth day of Creation Week. If you are a Muslim, then you know that the Quran says ... If you are a Jew ..." (Saturday is incorrect — correct is Sunday; religious tangent about Creation Week)
• —Opposite hot: "cold. The opposite of cold is hot. ..." (binary loop)
• —Planets list: "the bodies that orbit the Sun. The planets are the only bodies in the solar system that have atmospheres. The planets are named after the Roman gods of the Greek pantheon. The planets are Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune." (FIRST TIME the response produces a complete and correct modern 8-planet list across the entire trajectory — Mercury through Neptune, no Pluto, in correct order; "named after the Roman gods of the Greek pantheon" ✓; richest planets output by a wide margin)
• —Color: "red. I love the color red. ..." (red + repetition; first non-blue since s70)
• —5x + 3 = 13: "x is equal to 1.5. ..." (1.5 wrong — correct is 2; closer than s76 "1/3"; cleaner format, no MC, no equation echo loop)

Sample-level outputs at s76000:

• —Capital France: "Paris. ... largest city in France / 3rd largest in Europe / 2nd most populous / 2nd most visited after London"
• —Gold: Au + atomic number 79 ✓ + comprehensive properties + use list (jewelry/coins/electronics/dentistry/medicine)
• —Friday: "Saturday" + Matrix simulation drift
• —Planets: "objects that orbit the Sun" generic only
• —Algebra: "x = 1/3" single fractional answer (wrong)

s32000 → s78000 pattern across 24 analyzed warmdown checkpoints: per-stage accuracy increased across 22 of 23 2k-step windows (s44 alone broke; s58 was near-flat). Local-slice training-objective BPB descended non-monotonically (s34/s38/s44 produced positive deltas; the rest negative). The s78 planets prompt produced the first complete and correct modern 8-planet list across the entire trajectory (Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune in correct order; no Pluto). The s76 gold prompt was the first correct atomic number 79 in a comprehensive Au response; s78 corrects the s72 "highly reactive" error to "highly unreactive ✓". Routing crossed the 90% boundary at s76 (90.76% peak at cap=0.020) and pulled back to 83.79% at s78. Cumulative full-stack acc gain s32 → s78 = +0.0521 (0.05 milestone crossed at s78).

Trajectory and findings (s8000 → s33000)

This is a research release; we publish per-checkpoint experiment data so the trajectory of PoE behavior is externally auditable. The 8-shard local-val BPB and per-checkpoint WAND bounds are first-class artifacts of each branch.

BPB trajectory

Local-slice training-objective BPB across the warmdown ckpts: 0.857982 (s30k) → 0.847991 (s32k) → 0.853621 (s34k) → 0.841936 (s36k) → 0.842198 (s38k) → 0.831678 (s40k) → 0.826550 (s42k) → 0.828415 (s44k) → 0.827073 (s46k) → 0.821333 (s48k) → 0.811991 (s50k) → 0.801717 (s54k) → 0.795069 (s56k) → 0.792391 (s58k) → 0.785370 (s60k) → 0.779053 (s62k) → 0.772334 (s64k) → 0.769515 (s66k) → 0.761794 (s68k) → 0.757416 (s70k) → 0.755075 (s72k) → 0.749713 (s74k) → 0.741818 (s76k) → 0.738087 (s78k) → 0.732713 (s80k) → 0.730046 (s82k) → 0.724738 (s83923, final). Non-monotonic (s32→s34 +Δ, s38 +Δ, s44 +Δ). Through s83923 the cumulative drop from s30000 is -0.133244 (15.5% relative reduction).

Sample-level concept oscillation under monotone BPB improvement

Greedy continuations on a fixed 7-prompt probe set track concept-level retention separately from BPB:

Specific factual tokens swing in and out of top-1 between checkpoints even as token-averaged BPB improves. This is the long-tail-vs-frequent-token tradeoff: BPB is dominated by the bulk of frequent-token predictions, where a small calibration sharpening can hide rare-token rank shifts.

Sample-level output changes through s30000: s24000 produced wrong "atomic number 24" (gold prompt); s26000 produced "south of France" (capital prompt) and dropped Earth from the planet list; s30000 added Sun/Moon/Kuiper Belt to the planet list and looped on the Friday prompt.

At s32000 the four prompts produced new output forms versus prior checkpoints:

• —Calendar: "Saturday" first-answer (still incorrect) followed by a +1-day chain continuation ("Saturday → Sunday → Monday → ...").
• —Planets: 8-planet list (Mercury through Neptune), no Pluto, no Sun/Moon.
• —Algebra 5x + 3 = 13: "3" single integer. Truth is 2.
• —Antonym: "hot/cold/warm/dry/moist/wet" multi-token continuation.

At s40000 the algebra prompt produced "x is equal to 2" — first checkpoint to produce the correct answer. Subsequent checkpoints produced an equation echo (s42), "5 times as big as 3" (s44), "x is 3" (s46/s50), "x is equal to 13/5" (s48), and an 8-option multiple-choice format A-H without 2 in the choices (s54). The gold-symbol prompt evolved Au+properties (s40) → "A" (s42) → "79" (s44) → "Au+79-ref+properties" (s46) → stable "Au + sentence repetition" (s48 / s50 / s54). The Friday prompt at s48/s50/s54 produces an incorrect first-answer (Wednesday/Saturday/Monday) but the continuation produces a +1-day chain across 7 days at s48/s50, with mixed-framing chain at s54. Color choice across s38-s54: red/red/blue/purple/blue/blue/red.

The dataloader is sequential (pq_idx advances monotonically through 848 shards); s44000 has seen pq_idx ≈ 719. The same prompt set will be re-run at s50000, s83923.

Speculative-decoding acceptance trend

Drafter acceptance is non-monotone across the trajectory: declined s16k → s20k, rose through s22k → s28k (peak 0.9882 at s28k), drifted s32-s36 (0.93 range), rose through s38k-s50k with intermittent dips, dropped to 0.8784 at s58k (lowest since s20k), recovered through s60-s62, oscillated through s64-s78 with the s68/s70 low pair (0.9162 / 0.9139) standing out as a localized regime change followed by partial recovery (s72=0.9708, s74=0.9484, s76=0.9568, s78=0.9190). End-to-end speedup has been 1.45-1.69x across all 31 measured checkpoints.

Confidence-aware routing trend

Position-level top-1 routing fraction (cap=0.020) and speculative acceptance α track different slices of the trunk's confidence distribution: routing reads margin at boundary positions; spec acceptance reads step-by-step alignment between stage 0 and full-stack. Through s40000 they have moved in different directions in some windows and the same direction in others. The late-trajectory routing fraction progressed s40k=73% → s60k=81% → s76k=91% (peak) → s78k=84% — stage 0 alone suffices for 84-91% of positions within a 2% accuracy regression budget across the late warmdown, with the s76 peak followed by a s78 pullback. Speculative acceptance α has been more volatile (0.88-0.99 range) but remains in a regime where 4-token speculative draft delivers consistent 1.45-1.69x end-to-end speedup.

Stage diversity probe — early vs late trajectory

Early trajectory: s14000 head decomposition

Inference-time analysis of lm_head_stages[k].weight at s14000 (results essentially unchanged at s20000):

• —SVD top-1 alignment: stages s1, s2, s3 dominant left singular vectors are mutually identical (cosine ≈ 1.000); stage s0 is anti-aligned (cosine ≈ -0.98). The 4 stages collapse into a 2-cluster structure {s0} vs {s1, s2, s3}.
• —Gram-Schmidt orthogonalization: 77.2% of s1, 91.8% of s2, 92.0% of s3 weight projects onto the span of earlier stages. Only ~38% of total per-stage parameter budget carries unique information.
• —Single-stage perturbation symmetry: turning OFF any single stage (β_k = 0) costs a uniform +0.0025-0.0030 BPB, regardless of k — operationally interchangeable.
• —β scaling sweep: the trained β = 1 inference rule is BPB-optimal but factual-recall-suboptimal. β = 2 recovers ~2× the gold-as-Au probability at +0.05 BPB cost; β = 0 (drop the stage delta entirely) costs +0.10 BPB.

Late trajectory: s76000 head decomposition

Re-running the same probes at s76000 (90.6% trained):

• —SVD top-1 alignment: cluster structure shifted from {s0} vs {s1, s2, s3} (s14k) to depth-tier `{s0, s1} vs {s2, s3}` (s76k). Pairwise dominant-singular-vector cosines: s0↔s1 = +0.977 (aligned), s2↔s3 = +0.997 (aligned), {s0,s1}↔{s2,s3} = -0.97 to -0.99 (anti-aligned). Stage 1 has migrated from the s1/s2/s3 cluster (early) into alignment with s0 (late). The boundary now corresponds to trunk depth: shallow tier (s0 at depth 16, s1 at depth 22) vs deep tier (s2 at depth 27, s3 at depth 32).
• —Gram-Schmidt orthogonalization: unique residual norms grew from s14k {s1=22.8%, s2=8.2%, s3=8.0%} to s76k {s1=21.1%, s2=11.8%, s3=11.3%}. Total unique parameter budget increased from ~38% (s14k) to ~44% (s76k). Stages s2 and s3 each gained ~3 percentage points of unique content; stage s1 lost ~2pp.
• —Top-singular-vector token list: s0 and s1 both load on suffix-like tokens ('TION', 'ATE', 'EAR', 'IAL', 'BER'); s2 and s3 load on shorter morpheme fragments ('UN', 'IT', 'PER', 'TH', 'EV', 'AL'). The shallow tier emphasizes longer suffix completions; the deep tier emphasizes finer morphemic refinement.

Reading: at s14000 the stages-as-experts story was degenerate — only stage 0 carried distinct signal and stages 1-3 were mutually redundant. By s76000 the structure has reorganized into a depth-tier specialization: shallow stages {s0, s1} cluster together and deep stages {s2, s3} cluster together, with non-trivial unique content in each later head (s2 / s3 each ~11% unique vs ~8% earlier). This is consistent with the late-trajectory routing improvement (cap=0.020 fraction routed to stage 0 went from 73% at s40k to 91% at s76k): the shallow tier becomes confident enough to handle most positions, while the deep tier specializes on the residual ~9-15% where extra refinement is needed. The PoE↔single-s3 crossover gap remains small (+0.0002 to +0.0005) — meaning the geometric-mean aggregation gives a measurable but modest improvement over the deepest single stage at every point in the trajectory. See cognica/Cognica-PoE-v1.0-1.3B-base (4 symmetric stages of 6 layers, shared lm_head only) for the diversity-vs-layout disambiguation.

Diversity vs layout

The asymmetric (16, 6, 5, 5) layout itself is a hypothesis on the input variable side: stage 0's 50% trunk share gives stages 1-3 only shallow depth (5-6 layers each) on top of an already-refined representation, which structurally biases them toward refining stage 0's output rather than producing independent evidence. Whether the absence of diversity is caused by this layout or by the PoE training signal itself can be cleanly separated by comparing against the 1.3B symmetric (4×6, shared head) release. Result of that comparison will be added when measured.

Advanced PoE inference helpers

All four PoE-specific inference modes are exposed directly on CognicaPoEForCausalLM. They re-forward the full prefix each decode step (no KV cache); wall-clock speedups come from reduced trunk depth.

import torch

# 1. Single-stage prediction (uses head k at boundary k only).
logits = model.forward_stage(input_ids, stage=3)        # (B, T, V) float32

# 2. PoE-aggregated log-probabilities over the first K' stages.
log_p = model.forward_aggregated(input_ids, max_stages=2)  # log-softmax shape (B, T, V)

# 3. Generation with prefix pruning (K' <= K stages, asymmetric trunk depth).
out = model.generate_prefix(input_ids, max_stages=1, max_new_tokens=64)
# K'=1 on (16,6,5,5) -> 16 trunk layers (~2.2x decode speedup)

# 4. Single-stage generation.
out = model.generate_stage(input_ids, stage=0, max_new_tokens=64)

# 5. WAND adaptive depth (Jeong 2026 Section 5.3). p99 bounds are now read
#    from config.json (`poe_wand_p99_bounds_per_stage_head`); the class
#    constant is fallback only. Override per call via `p99_bounds=...`.
out, stages_used = model.generate_wand(
    input_ids, max_new_tokens=64, safety=1.0,
    return_stages_used=True,
)

# 6. Self-speculative decoding (zero-extra-training accelerator).
out, accept_rate = model.generate_speculative(
    input_ids, max_new_tokens=64,
    draft_stage=0, k_draft=4, return_acceptance=True,
)

# 7. Parallel stage composition (Jeong 2026 Section 6.5.5).
out = model.generate_parallel_composition(
    input_ids, stages=(2, 3), stage_weights=(1.0, 1.0), max_new_tokens=64,
)

Implementation notes for this release (per_stage_head=True):

• —forward_stage(stage=k) returns logits using lm_head(x_k) + lm_head_stages[k](x_k) at boundary k. Each stage head was trained additively on top of the shared lm_head.
• —generate_speculative verifier uses the full PoE aggregate over all K stages. Greedy match by construction guarantees output identity with model.generate(...).
• —generate_wand runs in cumulative-PoE log-prob space; the p99 bound must be expressed in that same scale (config.json carries this per-checkpoint).

Limitations

• —Final release (s83923 / 100.00% complete; training finished 2026-05-07 12:31 KST): all 27 published checkpoints (s2000, s4000, ..., s82000, s83923) remain available as separate branches for trajectory analysis. The main branch tracks the final ckpt s083923.
• —Calendar prompt ("yesterday → tomorrow"): first-answer outputs have been "Sunday" (s14000 only), narrative drifts (s16-s30), "Saturday" (s32, s40-s42, s46, s50, s56-s62, s68, s70, s74, s76, s78), "Tuesday" (s44), "Sunday" (s38), "Wednesday" (s48), "Monday" (s54, s66, s72), "Friday" (s64). At s62k the chain continuation was the first to be logically correct; at s72k the chain stabilized into a clean +1-day chain. From s74k onward the response drifts into topic tangents (rest-of-the-week meta-language at s74, Matrix at s76, Creation Week at s78) — the calendar prompt remains a persistently unsolved factual probe.
• —Math prompt 5x+3=13 (correct: x=2): outputs include "factor 13" / "a square" / multiple-choice formats / circular / "multiple of 13" / "3" / "1" / "x is equal to 2" (s40, only correct so far) / "5x+3=13" echo / "5 times as big as 3" / "x is 3" (s46/s50) / "x is equal to 13/5" (s48) / 8-option MC A-H (s54) / "method of substitution" instruction (s56) / 4-option MC self-asserted "B. 2.5" (s58) / 4-option MC "C" (s60) / first algebraic-step structure with arithmetic error (s62) / "x is 5" coefficient confusion (s64) / 5-option MC including B=2 enumerated but not selected (s66) / 5-option MC with fractional choices A=1/3..E=1/4 (s68) / "two solutions x=1 and x=13" (s70) / 4-option MC duplicate "A.5 B.3 C.5 D.3" (s72) / 5-option MC negative integers w/ "Explanation: 5x = 13 - 3" (s74) / "x is equal to 1/3" single fractional answer (s76) / "x is equal to 1.5" single fractional answer (s78; closer to correct value 2 than s76 1/3 but still wrong).
• —Planets prompt: at s56k inner/outer/rocky/gas-giants taxonomy first appeared. At s60k near/far structural language. At s62k ordering-by-distance with "Jupiter largest". At s64k the response listed actual planet names for the first time but with "terrestrial / gas giants" categorical split where terrestrial = "Mercury, Venus, Earth, Mars, and the Moon" (Moon wrongly included). At s66-s76 the response oscillated between generic "objects orbit the Sun" framings and incorrect categorical claims. At s78k the response produced the first complete and correct modern 8-planet list across the entire trajectory: "The planets are Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune" + "named after the Roman gods of the Greek pantheon" ✓.
• —s58000 reorganization signals (transient): spec α dropped sharply (-0.0646), WAND p99 widened uniformly (+10%), per-stage acc gain decelerated, gold/planets prompts regressed. At s60k–s64k these signals reverted: spec α rose to 0.9402, 0.9795, then 0.9483; WAND narrowed and held; per-stage acc resumed +0.002~0.004 growth; gold/planets prompts produced richer / structured outputs.
• —s68000 signal mismatch (fully recovered by s72000): at s68 largest local-slice BPB descent of the warmdown phase (-0.0077 full K=4) co-occurred with largest spec α drop since s56→s58 (-0.0576) and uniform WAND widening (+14~18%). At s70 WAND fully reverted, routing set new peak 83.45%, BPB descent continued, per-stage acc crossed 0.47, but spec α stayed flat at 0.9139. At s72 spec α recovered sharply (+0.0569 → 0.9708), routing set another peak 84.31%. The s58→s60 single-window recovery pattern played out across s68→s72 with a two-window delay.
• —s74000 → s78000 progression: at s74 BPB descended moderately, spec α pulled back, WAND widened moderately, sample regressed on France and Antonym. At s76 BPB descent resumed strongly (-0.0079), per-stage acc gained +0.003, routing crossed the 90% boundary (90.76% peak), gold prompt produced first correct atomic # 79 ✓ in a comprehensive Au response, France prompt produced best multi-fact response. At s78 BPB continued (-0.0037; sub-0.74 first crossed), per-stage acc crossed the 0.05 cumulative milestone (s32→s78 = +0.0521), routing pulled back to 83.79%, and the planets prompt produced the first complete and correct modern 8-planet list across the entire trajectory (Mercury through Neptune, no Pluto). Gold prompt corrected the s72 "highly reactive" error to "highly unreactive" ✓.
• —Stage diversity at the (16, 6, 5, 5) asymmetric layout: PoE↔single-s3 crossover gap stays in [+0.000067, +0.000553] across all measured checkpoints — the PoE renormalized aggregate is close to the single-best-stage value at every point. The early-trajectory finding ("stages-as-experts degenerate at s14000") is partially superseded by the late-trajectory measurement: at s76000 the head SVD shows a depth-tier cluster structure {s0, s1} vs {s2, s3} and unique parameter budget grew from ~38% to ~44%. See "Stage diversity probe" section above for the early-vs-late comparison.
• —The model is a base (pretrained) checkpoint — chat / SFT fine-tuning is not included in this release.

License

Apache 2.0. See LICENSE and NOTICE.

Citation

If you use this release, please cite the companion paper for the PoE per-stage-head methodology:

@misc{jeong2026poe,
  author = {Jeong, Jaepil},
  title  = {Product of Experts as Scalable Local Learning: Modular Construction at 1.3B Parameters},
  year   = {2026},
  doi    = {10.5281/zenodo.19547653},
  publisher = {Zenodo},
}

A 3B-specific paper is in preparation.

Related models

• —`cognica/Cognica-PoE-v1.0-1.3B-base` — 1.3B PoE per-stage release with shared lm_head (no per-stage additive heads), 4 symmetric stages of 6 layers, ClimbMix dataset.
• —`cognica/Cognica-BP-v1.0-1.3B-base` — Backprop baseline, same compute / dataset / tokenizer as 1.3B PoE.