Modelpublic

BananaMind-Model-Previewers/BananaMind-2-Micro-Preview

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes67downloads

Model Card

BananaMind 2 Micro Preview - 100%

This is the 100% training preview of BananaMind 2 Micro, a 2,933,193-parameter base causal language model. It is not an instruction-tuned chat model.

Architecture

Field	Value
Parameters	2,933,193
Layers	9
Hidden size	128
Intermediate size	512
Query/KV heads	4 / 2
Head dimension	32
Context	4,096
Vocabulary	2,048, tied
Attention	GQA, pre-RoPE QK norm
MLP	SwiGLU
Refresh	detached attention signal, kernel-9 causal depthwise gate, embedding reinjection

The learned refresh scalar is initialized to zero. During cached generation, the causal convolution history is carried alongside each layer's K/V cache.

Training

Field	Value
Progress	100%
Tokens seen	74,998,349,824
Target tokens	75,000,000,000
Hardware	8 x NVIDIA RTX PRO 6000 Blackwell Server Edition
Matrix optimizer	Stock `torch.optim.Muon`
Muon LR	0.05
Embedding/1D optimizer	AdamW
AdamW LR	0.004
Precision	bfloat16 autocast

Token range	FineWeb-Edu	DCLM	Cosmopedia v2	FinePhrase	FineMath	NPSet2
0.00B-15.00B	55%	30%	6%	4%	4%	1%
15.00B-30.00B	48%	26%	9%	8%	7%	2%
30.00B-48.75B	42%	20%	12%	12%	11%	3%
48.75B-63.75B	36%	16%	14%	15%	15%	4%
63.75B-75.00B	30%	12%	15%	18%	21%	4%

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind-Model-Previewers/BananaMind-2-Micro-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)