veyra-ai/Veyra2-Blueberry-5M-Base

Veyra2-Blueberry-5M-Base

Veyra2-Blueberry-5M-Base is a 5M-parameter Llama-like causal language model trained from scratch on approximately 30B tokens. It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation.

Model Details

Tokenizer

Special tokens:

• —<|eos|>: 0
• —<|bos|>: 1
• —<|im_start|>: 2
• —<|im_end|>: 3
• —<|pad|>: 4

Training Data

The model was trained on a 30B-token pretraining mixture.

30,000,000,000 tokens 120 shards

Mixture:

finewebedu: 40% finephrase: 20% dclmbaseline: 20% finemath_4plus: 20%

Training Summary

• —Final step: 14,305
• —Tokens seen: 30,000,000,000
• —Tokens per step: 2.097M
• —Sequence length: 2048
• —Last train loss: 2.95

Usage

<pre><code>import torch from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "veyra-ai/Veyra2-Blueberry-5M-Base"

tokenizer = AutoTokenizer.frompretrained( modelid, )

model = AutoModelForCausalLM.frompretrained( modelid, torchdtype=torch.float16, devicemap="auto", )

prompt = "In the 19th century" inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.nograd(): output = model.generate( **inputs, maxnewtokens=120, dosample=True, temperature=0.6, topp=0.9, repetitionpenalty=1.1, usecache=True, padtokenid=tokenizer.padtokenid, eostokenid=tokenizer.eostoken_id, )

print(tokenizer.decode(output[0], skipspecialtokens=False)) </code></pre>

Notes on Generation

Veyra2-Blueberry-5M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant. Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant.

Intended Use

This model is intended for:

• —small language model research
• —continued pretraining
• —benchmarking
• —experimentation with compact causal LMs

Limitations

• —Not instruction tuned
• —Not RLHF tuned
• —Not safe for factual or high-stakes use without additional validation
• —Can hallucinate names, citations, species, references, and technical claims
• —Open-ended text may drift off-topic
• —Context length during training was 2048 tokens

Citation

If you use this model, please cite the model repository:

veyra-ai/Veyra2-Blueberry-5M-Base