BananaMind/BananaMind-2-Pro-Preview
BananaMind-2-Pro-Preview
BananaMind-2-Pro-Preview is the first public checkpoint preview of BananaMind 2 Pro, a decoder-only base causal language model trained from scratch by BananaMind. This checkpoint was captured after 96,000 completed optimizer steps and 51,904,512,000 training tokens in an ongoing 100B-token pretraining run.
The model has 138,971,520 parameters, a 3,072-token context window, and a custom 32,768-token digit-aware byte-level BPE tokenizer. It uses grouped-query attention, QK normalization, RoPE, SwiGLU, RMSNorm, tied input/output embeddings, and a KV cache for generation.
This is a base model, not an instruction-tuned or chat model. Use continuation-style prompts and load the repository with trust_remote_code=True.
Preview Status
Benchmark scores describe this exact 96K preview checkpoint. They should not be treated as final BananaMind 2 Pro results.
Model Details
Evaluation
The BananaMind 2 Pro scores below were measured on the exported 96K checkpoint. ARC Easy, ARC Challenge, PIQA, and HellaSwag use acc_norm,none; ArithMark 3 uses length-normalized continuation accuracy; ArithMark 2 uses raw continuation accuracy. INT Index uses the Open SLM Leaderboard-style aggregate. Code Only is the Base Bench 1.1 code-completion category Elo, while Base Bench 1.1 reports overall fixed-item Elo.
The preview row is bold for emphasis; the strongest reported score in each metric is also bold. Base Bench comparison values for GPT-X2-125M, GPT-X-125M, SmolLM-135M, BananaMind-2-Medium, and GPT-2 are taken from the BananaMind Base Bench leaderboard. Metrics without a supplied or leaderboard result are marked N/A.
Code Only (Base Bench 1.1): 1295 Elo | 38/50 correct (76.00%) | 75.48% weighted accuracy
INT Index vs Training Compute
Comparison-model training compute is estimated as 6 x parameters x training tokens, matching the referenced GPT-X2 chart methodology. The Pro Preview point uses the supplied run estimate rather than recomputing it with the comparison approximation. GPT-2 and Pythia are excluded.
Base Bench Checkpoint Progression
The progression series is a consistent sweep over 24 exported checkpoints using CUDA, bfloat16, batch size 1, and the complete 350-item Base Bench 1.1 split. The 96K point in this sweep is 1105 Elo with 227/350 correct; the primary comparison and category tables use the separate CPU float32 result of 1106 Elo with the same 227/350 raw accuracy.
Base Bench Category Results
Evaluation results can vary with harness version, tokenizer handling, dtype, and scoring configuration. The published values are self-reported checkpoint evaluations.
Tokenizer
BananaMind-2-Pro-Preview uses a custom 32,768-token byte-level BPE tokenizer trained on 75 GiB of representative FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath-4+, and NPSet-2 Python educational data. It uses NFKC normalization and digit-aware pre-tokenization.
Digits are isolated before byte-level BPE so complete numbers are not merged into large number tokens.
Special token IDs:
Training Data
The ongoing 100B-token curriculum combines educational web text, broad web text, synthetic textbook material, mathematics, and Python educational code. The table describes the full-run target allocation; this preview was exported after 51.904512B tokens.
The run uses a capacity-aware curriculum:
Training Setup
Energy and Carbon Estimate
The following is an engineering estimate for training through this 96K preview checkpoint, not a wall-meter measurement. Runtime is derived from 51,904,512,000 tokens at the observed run-average throughput of 52,438 tokens/s. Only GPU and CPU package power were measured; all other component power, PSU loss, electricity-use, and emissions figures are estimates.
The grid factor is a recent Austrian daily consumption-based estimate from Electricity Maps. Applying its reported 2024 and 2025 flow-traced annual means of 125.5 and 169.4 gCO2e/kWh to the same energy estimate gives 13.5-18.2 kg CO2e. This estimate excludes embodied hardware emissions, the display, and external networking or storage infrastructure.
Usage
Install the runtime dependencies:
pip install -U torch transformers safetensorsLoad the model with custom architecture code enabled:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "BananaMind/BananaMind-2-Pro-Preview"
tokenizer = AutoTokenizer.from_pretrained(
model_id,
trust_remote_code=True,
)
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
torch.bfloat16
if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
else torch.float32
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
dtype=dtype,
).to(device).eval()
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=80,
do_sample=True,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.1,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))For deterministic continuation scoring, use do_sample=False. For free-form sampling, a temperature of 0.6 to 0.8, top_p=0.9, and repetition_penalty=1.1 are reasonable starting points.
Intended Use
BananaMind-2-Pro-Preview is intended for base-model research, local experimentation, text continuation, tokenizer research, arithmetic evaluation, checkpoint analysis, and small-language-model comparisons.
It is not instruction-tuned and does not use a chat template. It has not received dedicated safety alignment and may produce incorrect, biased, repetitive, or otherwise undesirable text. Do not rely on its output for high-stakes decisions.
License
This repository is released under theBananaMind Community License 1.0. Commercial products or services exceeding either threshold in Section 1 require a separate commercial license from Banaxi-Tech.