BananaMind/BananaMind-2-Pro

[image]

BananaMind-2-Pro

BananaMind-2-Pro is a 138,971,520-parameter decoder-only base language model trained from scratch by BananaMind. The completed pretraining run processed 99,999,449,088 tokens—the scheduled realization of a 100B-token curriculum—over 184,954 optimizer steps.

The model has a 3,072-token context window and a custom 32,768-token digit-aware byte-level BPE tokenizer. Its architecture uses grouped-query attention, QK normalization, RoPE, SwiGLU, RMSNorm, tied input/output embeddings, and a KV cache for generation.

This is a base model, not an instruction-tuned or chat model. Use continuation-style prompts and load the repository with trust_remote_code=True.

Release Status

Model Details

Architecture Overview

The model maps tokens through a tied input embedding, 24 pre-normalized Transformer blocks, a final RMSNorm, and the tied language-model head. Each block contains causal grouped-query self-attention followed by a SwiGLU feed-forward network, with residual connections around both sublayers.

Eight query heads share four key/value heads, reducing KV-cache size while retaining multi-head query capacity. QK normalization stabilizes attention logits, and RoPE supplies position information without learned absolute-position embeddings. During autoregressive generation, each layer can reuse cached keys and values instead of recomputing the full prefix.

Evaluation

[image]

These results describe the exact final checkpoint at step 184,953. ARC Easy, ARC Challenge, PIQA, and HellaSwag use zero-shot acc_norm,none. ArithMark 3 uses length-normalized continuation accuracy, while ArithMark 2 uses raw continuation accuracy. Code Elo is the Base Bench 1.1 code-completion category; Base Bench 1.1 Elo is the complete 350-item suite.

The INT Index chance-normalizes HellaSwag, the mean of ARC Easy and ARC Challenge, PIQA, and ArithMark 3. ArithMark 2 is not included:

N(s, c) = 100 * (s - c) / (100 - c)
INT = [N(H,25) + N((ARC_E+ARC_C)/2,25) + N(P,50) + 0.65*N(A3,25)] / 3.65

INT Index vs Training Compute

[image]

Training compute is estimated consistently as 6 × parameters × training tokens. Using the exact 138,971,520-parameter count gives 83,382.91 PFLOPs for Pro Final at 100B tokens and 43,279.49 PFLOPs for Pro Preview at 51.904512B tokens. GPT-X2.5 uses 135M parameters, 75B tokens, and a 2,048-token context window; context length is not a separate term in this 6ND approximation.

Base Bench Checkpoint Progression

[image]

This progression contains 41 complete 350-item evaluations from 2.70B tokens through the final 99.999B-token checkpoint. Every point uses CUDA, bfloat16, and batch size 1 for consistency. The final batch-1 endpoint is 1132 Elo, 236/350 correct (67.43%), and 64.69% weighted accuracy. The primary comparison table above retains the separately measured batch-32 result.

Base Bench 1.1 Category Results

The final model passed 233 of 350 cases for 66.57% raw accuracy and 63.73% weighted accuracy. Its overall fixed-item rating is 1124 Elo.

Base Bench was evaluated locally with CUDA, bfloat16, and batch size 32. ArithMark 3 used CUDA bfloat16 with batch size 64; ArithMark 2 used CUDA bfloat16 with batch size 1. Results can vary with harness versions, tokenizer handling, dtype, and scoring configuration. All values are self-reported evaluations.

Tokenizer

BananaMind-2-Pro uses a custom 32,768-token byte-level BPE tokenizer trained on 75 GiB of representative FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath-4+, and NPSet-2 Python educational data. It uses NFKC normalization and digit-aware pre-tokenization.

Digits are isolated before byte-level BPE, preventing complete numbers from being merged into large number tokens.

Special token IDs:

Training Data

The 100B-token curriculum combines educational web text, broad web text, synthetic textbook material, mathematics, and Python educational code.

Training used a capacity-aware curriculum:

Training Setup

The schedule used a 2,000-step linear warmup, a long stable learning-rate region, and a final cosine decay over the last 15% of training. Weight decay dropped from 0.1 to 0.01 after 40B tokens, and the auxiliary z-loss was disabled at the same boundary. The final optimizer update completed the planned 99,999,449,088-token schedule without rounding up to an extra sequence.

Energy and Carbon Estimate

The following is an engineering estimate for the complete pretraining run, not a wall-meter measurement. Runtime is extrapolated from the observed 52,438-token/s run average. GPU and CPU package power were measured during training; motherboard, memory, storage, PSU loss, electricity use, and emissions are estimates.

Using the cited 2024 and 2025 Austrian flow-traced annual means of 125.5 and 169.4 gCO2e/kWh gives an estimated range of 26.0–35.1 kg CO2e. The grid figures come from Electricity Maps. This estimate excludes embodied hardware emissions, tokenizer training, evaluation, external networking, and remote storage.

Usage

Install the runtime dependencies:

pip install -U torch transformers safetensors

Load the custom architecture with remote code enabled:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-Pro"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

For deterministic continuation scoring, use do_sample=False. For free-form sampling, a temperature from 0.6 to 0.8, top_p=0.9, and repetition_penalty=1.1 are reasonable starting points.

Intended Use

BananaMind-2-Pro is intended for base-model research, local experimentation, text continuation, tokenizer research, arithmetic evaluation, checkpoint analysis, and small-language-model comparisons.

Suitable uses include studying training dynamics, testing compact-model architectures, comparing tokenizers, measuring continuation likelihoods, and using the checkpoint as a starting point for task-specific fine-tuning.

Limitations

• —This is a base model and does not reliably follow conversational instructions.
• —The 3,072-token context window limits long-document use.
• —At 139M parameters, factual recall and multi-step reasoning remain capacity-constrained.
• —The training mixture is English-heavy, so performance in other languages is not characterized.
• —The model has not received dedicated safety alignment and can produce inaccurate, biased, repetitive, or undesirable text.
• —Loading requires repository-provided custom Transformers code with trust_remote_code=True.

Do not rely on the model for medical, legal, financial, safety-critical, or other high-stakes decisions.

License

This repository is released under the BananaMind Community License 1.0. Commercial products or services exceeding either threshold in Section 1 require a separate commercial license from Banaxi-Tech.