Modelpublic

Banaxi-Tech/pico-80

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes53downloads

Model Card

BananaMind 2 Pico Test - 80%

This is the 80% checkpoint of a 900,002-parameter base causal language model. It is not instruction tuned.

Architecture

Field	Value
Parameters	900,002
Layers / hidden size	6 / 96
SwiGLU intermediate size	380
Query / KV heads	6 / 2
Head dimension	16
Context	4,096
Vocabulary	384, tied
Refresh layers	4 and 6
Refresh kernel	Causal depthwise, width 9

The selective XSA refresh gate reads detached attention output as its signal, reinjects the original input embedding as its value, and carries convolution history alongside the K/V cache. Its learned residual scalar starts at zero.

Training

Field	Value
Progress	80%
Tokens seen	160,004,308,992
Target tokens	200,000,000,000
Hardware	4 x NVIDIA H200
Matrix optimizer	Stock `torch.optim.Muon`
Muon peak LR	0.07
Embedding/control optimizer	AdamW, LR 0.004
Precision	bfloat16 autocast

Token range	FineWeb-HQ	Cosmopedia v2
0.00B-100.00B	80%	20%
100.00B-200.00B	60%	40%

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Banaxi-Tech/pico-test"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)