BananaMind/BananaMind-2.1-Unified

[image]

BananaMind-2.1-Unified

BananaMind-2.1-Unified is a three-tower decoder-only causal language model trained from scratch by BananaMind on a 38B-token flat mix inspired by the brain having two hemispheres. It is the successor experiment to BananaMind-2-Unified, and the first BananaMind model where the towers cannot talk to each other directly at all.

Three transformer stacks share one d=384 embedding. A and C are the outer towers and each owns an output head; the next token is a probability-space mixture of the two. B is the relay — it has no output head, no loss term of its own, and is the only path between A and C. Everything the outer towers share has to survive a trip through the middle, and B is trained entirely by gradient arriving through its four bridge directions.

The model has 34,999,041 parameters, a 4,096 token context window, and the same custom 8k-token digit-aware byte-level BPE tokenizer as BananaMind-2-Mini.

This is a base model. It is not instruction tuned.

[image]

Model Details

Architecture

The three towers

Parameter budget

Tower C reads the shared embedding natively at d=384; A and B get a linear input projection. The four bridge families total 1,232,640 parameters.

Exchange schedule

Three rounds, 1-indexed layer numbers. Bridge output is added to the residual before the receiving block, which guarantees the relay always has real processing between taking a signal in and handing one back out.

Placement is biased late on the outer towers because that is where the 2.0 run's gates actually grew. Note the two structural consequences: A's layers 13-14 run after the final bridge read and can never influence C, and C's layer 6 runs after B is finished and can never influence A.

Bridge gates

Every bridge is per-channel gated and initialised to 0.01 rather than 0. In 2.0 a zero init was correct because both towers had their own loss and the bridges were a bonus. Here B has no loss term, so a zero init risks the middle receiving no gradient on step 0 and never waking up.

It woke up. Mean |gate| in the final checkpoint, against the 0.01 init:

Every one of the twelve gates grew, by 4x to 47x. Two patterns are visible: traffic is strongly biased toward the deep rounds in three of the four directions (the same concentration 2.0 showed), and the into-B directions carry noticeably larger gates than the out-of-B ones, with B -> C the quietest channel in the model.

The mixer

The next-token distribution is a two-way probability-space mixture, not a logit sum:

alpha = sigmoid(mix_head([ln_f_a(h_a) ; ln_f_c(h_c)]))
log p = logaddexp(log alpha + log_softmax(lm_head_a(h_a)),
                  log(1 - alpha) + log_softmax(lm_head_c(h_c)))

alpha is the per-token weight on tower A. Measured on an 84-token mixed history/math/science passage: mean 0.433, range 0.049 to 0.865, with 41.7% of tokens weighted toward A. The mixer is genuinely token-dependent, not collapsed onto one head.

Tokenizer

Identical to the BananaMind-2-Mini tokenizer: a custom 8k byte-level BPE trained on FineWeb-Edu text with digit-aware pre-tokenization. Digits are kept as separate tokens so numbers do not collapse into large number tokens.

Digit IDs:

Examples:

18  -> [20, 27]
227 -> [21, 21, 26]

Special token IDs:

Training Data

38B tokens, streamed, flat mix, no curriculum ramp. 2.1 changes topology, and a moving data distribution on top of that would make the comparison against 2.0 unreadable. The shares are Mini's final aggregate targets held constant from the first token.

Dataset revisions are pinned in checkpoint_metadata.json.

Training Setup

Loss

L = L_mix + 0.3 * (L_A + L_C)

There is no `L_B` term. B trains entirely on gradient arriving through its four bridge directions.

Final training-batch losses at step 72,470:

These are training-batch numbers, not held-out. Note also that L_A and L_C are computed with the bridges live, so they do not predict standalone tower performance — that was the central misreading in the 2.0 run. The ablation modes below answer it properly, and the gap is enormous: solo head A logs 2.643 with the bridges live and 10.386 without them.

Full per-step history is in training_metrics.jsonl.

Evaluation

lm_eval 0.4.12, zero-shot, fp32 on one RTX 5070 Ti, --batch_size auto (settled at 64), full test sets (ARC-Easy 2,376 / PIQA 1,838 / HellaSwag 10,042). All task scores are acc_norm, matching the BananaMind-2-Mini card. Every ablation mode was evaluated on the same run.

Approximate standard errors: ARC-Easy ±1.0, PIQA ±1.1, HellaSwag ±0.5 points.

Against the previous generation on the same three tasks:

The three-tower model is roughly level with Mini overall — ahead on PIQA by 2.1 points, behind on ARC-Easy by 1.4 — for 39% more parameters and 27% more tokens. On aggregate benchmarks the three-tower model is roughly level with Mini. The contribution of this architecture is not a benchmark number — it is what the ablations and lens data reveal about how integration, specialisation, and understanding organise themselves when the only path between two output towers is a silent relay that has no voice of its own.

What the ablations show on benchmarks

• —Only the intact model is clearly above chance. full is +10.1 points over the chance average. Everything else falls between +0.7 and +6.5, and the bottom four modes sit within a few points of chance on all three tasks.
• —`a_only` is indistinguishable from chance (34.01 vs 33.33; PIQA 50.11 against a 50.00 floor). Tower A alone, with a 14-layer stack and its own trained head, has essentially no standalone ability.
• —Severing the bridges is no better than deleting two towers. cut_bridges (35.76) and c_only (35.91) are within noise of each other. Three towers that cannot communicate are worth no more than tower C on its own — which is what you would expect if C's head carries the model whenever the relay is dead.
• —A relay that only forwards recovers most of the gap. bypass_b (39.61) sits 3.9 points above cut_bridges while performing zero computation in the middle, and B's actual computation is worth a further 3.8 on top.

Where the benchmarks disagree with the loss

The NLL ablations put ab_only (4.845) well ahead of cb_only (7.373). The benchmarks reverse it: cb_only averages 39.84 against ab_only's 35.43. Both measurements are correct and they are measuring different things — NLL is absolute calibration over running text, while acc_norm is length-normalised ranking among a fixed set of candidate answers. A+relay predicts ordinary text more accurately; C+relay discriminates better between multiple-choice options. C reads the shared embedding natively at d=384 and owns the larger head (3.15M vs 2.10M), which is the likeliest explanation for the discrimination advantage.

The practical reading: do not treat either metric alone as "which tower matters more". They rank the halves of this model in opposite orders.

BananaMind Base Bench 1.1

Four-choice base-text continuation scored by mean conditional token log-probability, 350 cases across 7 categories, run per mode. Chance accuracy is 25%.

Per-category Elo (50 cases each, so single-category gaps under ~100 Elo are noise):

Two things here that the lm_eval table does not show:

• —`ab_only` (867) and `a_only` (866) are the same score. On this benchmark, giving tower A a live relay and a running tower B buys essentially nothing over running A completely alone. Whatever the relay contributes, it needs tower C at the other end of it — consistent with the lens finding that B's representation only becomes readable when both outer towers feed it.
• —The A/C ranking flips again. Base Bench puts the A-side modes above the C-side ones (ab_only/a_only 867/866 over cb_only/c_only 803/774), while lmeval `accnorm ranked them the other way (cbonly` 39.84 over `abonly 35.43). Three metrics have now ordered the two halves of this model three different ways — NLL favours A, acc_norm` favours C, Base Bench Elo favours A. None of them is wrong; "which tower matters more" is simply not a metric-independent question here.

full is the only mode meaningfully clear of chance on accuracy, and the only one above 900 Elo. Raw reports and per-case predictions for every mode are in eval_results/base_bench_1.1/<mode>/.

Jacobian lens across the modes

Fitted with Anthropic's jacobian-lens method: lens_l(h) = unembed(J_l @ h) with J_l = E[∂h_final / ∂h_l], averaged over 6 prompts and all valid source/target positions. For the two-headed modes the target basis is the joint vector z = [x_a ; x_c] (256 + 384 = 640), since the mixture is a deterministic function of that single vector and no individual tower's basis can express it. Single-head modes use that head's basis alone.

Influence mass

Scale-normalised mean ‖J‖_F per tower — each half divided by RMS · √d, because the towers have different widths and different residual scales, so raw Frobenius norms would just measure residual magnitude. The bracketed figure is the share of that tower's influence landing in C's half of the joint basis.

Three of these rows are self-validating. Under cut_bridges tower B's mass is exactly 0.000 — B owns no head, so with the bridges dead it cannot reach the output at all, and the lens recovers that from the gradients without being told. The same row shows A contributing 0.0% to C's half and C contributing 100% to it: the two outer towers are perfectly decoupled. And a_only (3.505) and c_only (5.873) reproduce the cut_bridges tower masses to the digit, which is the only thing they could do if bridge-cutting truly isolates the towers.

Lens readout at each tower's deepest block

Four things fall out:

1. 1.The relay carries the answer, and only in `full`. B has no head and no loss term, yet in full its deepest block reads Paris/ France, oxygen/ hydrogen, blue. This is the README's original "does B wake up?" question answered from the representation rather than from gate magnitudes.
2. 2.An orphaned B is not merely weak, it is unreadable. Under cut_bridges B's readout collapses to the four special tokens on every prompt — the residual of a stack that receives nothing and reaches nothing.
3. 3.*B needs both outer towers to become semantic.* Under ab_only and cb_only the bridges are live and B still runs, but its readout is junk ( High Tem Cal, dom vere ure). B's task content is not something A alone or C alone puts there; it appears only at the confluence.
4. 4.`bypass_b` sharpens the outer towers while making the model worse. With B reduced to the identity, tower A's own readout gets more directly predictive ( France French, versus the B in in full) and both tower masses rise sharply (A 3.572 → 5.138, C 8.370 → 13.205). The outer towers compensate by carrying more themselves — and still lose 3.8 points of benchmark average. What B computes is not replaceable by the outer towers working harder.

Caveats: this is a linearised, corpus-averaged sensitivity, not a causal contribution — a gate-ablation KL would be the confirming experiment. The normalisation is also a choice; raw Frobenius mass gives a different picture because tower A's final residual runs ~3.3x hotter than C's. Fit settings (6 prompts, 48 tokens, first 8 positions skipped) are lighter than a publication-grade run, so treat small differences between adjacent rows as noise.

What Tower B represents

The lens readout reveals something unexpected about how the relay processes information. Tracing B's representations layer by layer across multiple prompts shows a consistent pattern: B processes significance before facts.

The early layers (B-L0, B-L1) consistently produce evaluative and abstract terms — not factual content, not input echoes, but something closer to significance assessment. By the final layer (B-L4), B has arrived at the correct answer: Paris/France, Water/hydrogen, blue/green.

This trajectory — from evaluation to categorisation to answer — is consistent across prompts and mirrors the ordering of affective and cognitive processing observed in biological neural systems, where emotional evaluation precedes and shapes factual retrieval. No part of the architecture or training objective was designed to produce this ordering. B discovered it.

B's influence distribution across the two output towers is nearly perfectly balanced (49.1% toward C), confirming that B functions as a symmetric integration hub rather than favouring either side.

The PIQA result

The aggregate benchmark comparison with Mini understates what the ablations reveal about physical reasoning. PIQA measures physical intuition — understanding that you pour water into a cup, not a fork — and it is the benchmark where the relay topology produces its clearest separation.

Tower A alone scores 50.11% on PIQA: indistinguishable from the 50.00% chance floor. Tower C alone scores 52.12%: barely above chance. The full system scores 61.75%. The entire physical reasoning capability of this model — all 11.75 points above chance — is a product of integration. Neither output tower can reason about the physical world on its own.

This makes PIQA the sharpest measure of what the relay contributes. It is not a capability that either tower possesses and the relay merely enhances. It is a capability that exists only in the integration and nowhere else.

For comparison, the single-tower BananaMind-2-Medium at 50M parameters scores 59.41% on PIQA. The 35M three-tower model exceeds it by 2.3 points, with a silent middle tower that produces no output and consumes roughly 5.8M of the parameter budget on pure integration.

Emergent properties of integration

Three findings from the ablation and lens data point to integration as an emergent rather than additive phenomenon:

B needs both outer towers to become semantic. Under ab_only and cb_only, B's readout collapses to junk despite having live bridges and running its full computation. B's task-relevant representations — Paris, oxygen, blue — appear only when both A and C feed it simultaneously. This is not A's knowledge or C's knowledge routed through B. It is something new that exists only at the confluence.

Dedicated understanding parameters outperform general-purpose ones. Tower B has no output head and no loss term. Its 5.8M parameters are trained entirely by indirect gradient arriving through four bridge directions. Yet removing B (the bypass_b ablation) costs 5.82 PIQA points and 59 Elo on Base Bench. Those 5.8M parameters, freed from the output objective and devoted entirely to integration, contribute more per parameter than any equivalent allocation to the output towers could.

The channel matters, but so does the computation. bypass_b replaces B with the identity, keeping the bridges live but performing zero computation in the middle. This recovers 3.59 nats over cut_bridges, showing that a path between A and C is valuable even without processing. But B's actual computation adds a further 2.83 nats on top. The relay is not merely a conduit; it transforms what passes through it.

ArithMark-3

This one predates the sweep above: it was run against the unpinned main of the source repo on 2026-08-18, before the 100% checkpoint in this folder was pulled, so it is not confirmed to be this exact checkpoint, and it was run only in full mode. Treat it as indicative.

Repository Files

Usage

This model uses custom architecture code, so load it with trust_remote_code=True.

Install dependencies:

pip install -U transformers safetensors torch

Run inference:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "BananaMind/BananaMind-2.1-Unified"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

prompt = "The capital city of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

KV cache

Supported, and on by default — generate(), use_cache=True, and a manually supplied DynamicCache all work. Decoding is O(n) per token instead of a full three-tower re-forward.

A relay model has no single residual stack, so the three towers share one flat cache index space, in the order the config documents:

config.num_hidden_layers is 25, the sum of the three tower depths. It exists purely so generic Transformers tooling — DynamicCache above all — can size a per-layer cache. It is derived in the config class, never read back from a serialised config.json, so a stale value cannot desync it from the layer mapping. Each tower keeps its own entries; nothing is shared between them.

Two properties of the architecture make this work without any relay-specific machinery:

• —Bridges do not mix positions. An Edge is a per-channel gate on a linear map, so every bridge contribution for a newly arriving token is computable from that token's own tower states. No bridge output has to be cached alongside the KV states.
• —RoPE is translation-invariant in the attention logits. It enters only through the relative query–key offset, so shifting a whole sequence — which is exactly what left padding does — leaves every attention score unchanged. Absolute positions taken from the cache length are therefore correct for padded batches too.

Measured on CPU, fp32, from a 221-token prompt:

Verified equivalences: token-by-token cached decode reproduces the full uncached forward (max abs error 3.1e-05 on fp32 log-probs), chunked prefill in 5/4/5-token pieces reproduces it too, greedy generation is bit-identical with and without the cache over 48 tokens, and left-padded batched generation matches the same prompts run singly. cut_bridges=True was checked separately and also matches.

Scoring passes are unaffected: when labels is supplied, forward() skips cache construction, since a loss pass consumes the whole sequence at once and would otherwise allocate 25 layers of state for nothing.

What .logits contains

.logits holds a normalised log-probability vector, not unnormalised logits, because the two heads are mixed in probability space. Verified on this checkpoint: logsumexp over the vocabulary is 1.2e-06, i.e. zero.

log_softmax is the identity on it, so loglikelihood scoring, generate() and temperature-1 sampling all behave correctly. The one thing that is not meaningful is treating the numbers as unnormalised scores with an arbitrary additive offset — they are already calibrated.

Reading the towers individually

hidden_states() returns the two final tower residuals and the raw mixer logit, which is the entry point for any interpretability work on the relay:

h_a, h_c, mix_logit = model.hidden_states(input_ids)
alpha = torch.sigmoid(mix_logit)   # per-token weight on tower A

Ablation modes

Because A and C have no direct path to each other, "what is the relay worth?" is only answerable by cutting the model apart. relay_mode selects where to cut. All seven modes run on the same weights, with no retraining.

bypass_b is the interesting one: tower B's five blocks are replaced by the identity, but every bridge stays live. A and C still exchange signal — through a relay that does no computation. That separates "the relay computes something" from "a channel exists at all".

Three ways to select a mode, checked against each other so a contradiction raises rather than resolving by some invisible precedence rule:

# 1. at load time
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True,
                                             relay_mode="bypass_b")

# 2. the single-tower alias  ("a" or "c"; B has no head and cannot run alone)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True,
                                             use_single_tower="a")

# 3. per call, for sweeping without reloading
logits = model(input_ids, relay_mode="ab_only").logits
model.generate(input_ids, max_new_tokens=64, relay_mode="cb_only")

# or switch in place (start a fresh KV cache afterwards)
model.set_relay_mode("cut_bridges")

cut_bridges=True predates relay_mode and still works, selecting the cut_bridges mode.

In modes where a tower is off, hidden_states() returns None in that tower's slot, and mix_logit is None whenever only one head is live. In full mode all three are always tensors, so existing callers are unaffected.

Measured ablation results

Mean NLL over 12 held-out encyclopedic passages, 519 predicted tokens, fp32:

Four things fall out of this, and they are the answer to the question 2.0 left open:

1. 1.Neither outer tower survives alone. a_only and c_only both score worse than uniform over the 8,192-token vocabulary. Without its partner each head is not merely degraded, it is confidently wrong. 2.0's tower-B bridge dependence reappears here in the extreme, and on both towers at once.
2. 2.Three towers that cannot talk are barely better than guessing. cut_bridges at 8.669 sits just under the 9.011 uniform baseline.
3. 3.The channel is worth more than what runs inside it. Going from cut_bridges to bypass_b recovers 3.59 nats with tower B performing zero computation — merely existing as a path. B's actual computation is then worth a further 2.83 nats on top.
4. 4.The relay is not symmetric in value. ab_only (4.845) beats cb_only (7.373) by 2.5 nats, even though C is the wider tower and owns the larger head. A plus a computing relay, with C absent entirely, also edges out all three towers with a non-computing relay.

Note that this measures the mode's loss, not a benchmark score, on a small sample. It is a sharp instrument for relative comparison between modes and not a substitute for a full lm_eval sweep.

Suggested Generation Settings

For stable continuations:

• —do_sample=False
• —repetition_penalty=1.1
• —max_new_tokens=64 to 160

For more varied text:

• —do_sample=True
• —temperature=0.6 to 0.8
• —top_p=0.9
• —top_k=50
• —repetition_penalty=1.1
• —max_new_tokens=64 to 192

Findings

Intended Use

BananaMind-2.1-Unified is intended for multi-tower architecture research, relay and information-bottleneck experiments, interpretability work on cross-tower routing, workspace and integration research, consciousness research, lightweight language-model research, local experimentation, and text continuation.

Relationship to Other BananaMind Models

2.0 answered one question — two towers with cuttable bridges do beat either tower alone — and raised a sharper one. Its gate data showed traffic concentrated deep and heavily asymmetric, and the bench then showed one tower had traded away standalone ability for bridge dependence. 2.1 asks what happens when the exchange is not a side channel between two peers, but the only path between them.

We're releasing BananaMind-2-Unified soon when its done training.

The architecture, trainer, and export tooling live in BananaMind-2.1-Unified-Arch/.

License

Apache 2.0