veyra-ai/Veyra2-Apricot-50M-Base

Veyra2-Apricot-50M-Base

Veyra2-Apricot-50M-Base is a 49.3M-parameter Llama-like causal language model trained from scratch on approximately 20B tokens.

It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation. If you are looking for a base model that is better at prompt continuation you should use Veyra2 Mango 30M Base instead.

Model Details

Tokenizer

Special tokens:

• —<|bos|>: 0
• —<|eos|>: 1
• —<|pad|>: 2
• —<|unk|>: 3
• —<|im_start|>: 4
• —<|im_end|>: 5

Training Data

The model was trained on a 20B-token pretraining mixture.

• —FineWeb-Edu sample-10BT: 45%
• —Cosmopedia-v2: 30%
• —DCLM-Edu: 15%
• —FineMath finemath-3plus: 10%

Training Summary

• —Final step: 9532
• —Tokens seen: 19,990,052,864
• —Tokens per step: 2,097,152
• —Global batch size: 2048
• —Sequence length: 1024
• —Last train loss: 2.5062
• —Final eval loss: 2.4978
• —Final eval perplexity: 12.16

Evaluation

SciCloze Eval

• —SciCloze-900 accuracy: 47.56%
• —SciCloze-900 correct: 428 / 900

SciCloze-900 repository: veyra-ai/SciCloze-900

SciCloze-900 subject breakdown:

• —Biology: 53.67%
• —Physics: 45.67%
• —Chemistry: 43.33%

Other Evals

Zero-shot results:

Open SLM Bench verified:

• —Average: 38.81%
• —PIQA acc_norm: 62.13%
• —ARC-Easy acc_norm: 42.47%
• —ARC-Challenge acc_norm: 23.29%
• —HellaSwag acc_norm: 31.28%
• —ArithMark-3.0 acc: 31.60%

Additional local evaluations:

• —Winogrande acc: 50.12%
• —OpenBookQA acc_norm: 28.80%
• —BoolQ acc: 60.67%
• —SciQ accuracy: 72.70%
• —SciQ normalized accuracy: 64.20%

Usage

This model uses custom Transformers remote code because it includes QKV-Norm and a custom architecture definition.

<pre><code>import torch from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "veyra-ai/Veyra2-Apricot-50M-Base"

tokenizer = AutoTokenizer.frompretrained( modelid, trustremotecode=True, )

model = AutoModelForCausalLM.frompretrained( modelid, trustremotecode=True, torchdtype=torch.float16, devicemap="auto", )

prompt = "In the 19th century" inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.nograd(): output = model.generate( **inputs, maxnewtokens=120, dosample=True, temperature=0.6, topp=0.9, repetitionpenalty=1.1, usecache=True, padtokenid=tokenizer.padtokenid, eostokenid=tokenizer.eostoken_id, )

print(tokenizer.decode(output[0], skipspecialtokens=True)) </code></pre>

Notes on Generation

Veyra2-Apricot-50M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant. Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant.

Intended Use

This model is intended for:

• —small language model research
• —continued pretraining
• —benchmarking
• —science-oriented cloze and multiple-choice evaluation
• —experimentation with compact causal LMs

Limitations

• —Not instruction tuned
• —Not RLHF tuned
• —Not safe for factual or high-stakes use without additional validation
• —Can hallucinate names, citations, species, references, and technical claims
• —Open-ended text may drift off-topic
• —Context length during training was 1024 tokens

Citation

If you use this model, please cite the model repository:

veyra-ai/Veyra2-Apricot-50M-Base