veyra-ai/Veyra-30M-Base

Veyra 30M Base

It is not instruction tuned and should not be evaluated like a finished chat assistant. It is expected to hallucinate, repeat, fail simple factual/math prompts, and continue text in odd ways.

Training summary

Approximate training stages:

• —1B tokens: Cosmopedia v2 bootstrap pretraining.
• —+1.5B tokens: mixed continuation using Cosmopedia-v2 repository configs including cosmopedia-v2, fineweb-edu-dedup, and python-edu.
• —+2.5B tokens: Went back to Cosmopedia v2 but increased context length from 512 -> 1024.
• —Total: about 5B pretraining tokens.

Architecture

Veyra-30M is a small attention-sparse decoder-only language model.

Key details:

• —Parameters: 34.6M
• —Vocabulary: 8,192 tokens
• —Hidden size: 512
• —Layers: 8
• —Attention heads: 8 query heads, 2 KV heads
• —MLP intermediate size: 2048
• —Activation: SwiGLU
• —Normalization: RMSNorm
• —Position encoding: RoPE
• —Tied token embeddings / LM head
• —Context in this checkpoint: 1024 tokens

Loading

This repository uses custom Transformers code.

Minimal usage:

from transformers import AutoTokenizer, AutoModelForCausalLM import torch

repo = "veyra-ai/Veyra-30M-Base"

tokenizer = AutoTokenizer.frompretrained(repo, trustremotecode=True) model = AutoModelForCausalLM.frompretrained(repo, trustremotecode=True, dtype=torch.float32) model.eval()

prompt = "Photosynthesis is the process by which" inputids = tokenizer.encode(prompt, addspecialtokens=False, returntensors="pt")

with torch.nograd(): out = model.generate( inputids, dosample=True, temperature=0.5, topk=30, repetitionpenalty=1.15, norepeatngramsize=2, maxnewtokens=80, )

print(tokenizer.decode(out[0], skipspecialtokens=True))

For raw completion prompts, use add_special_tokens=False.

Optimizer

Training used:

• —CosineGatedAdam / CGA-v0 on 2D projection matrices
• —AdamW on embeddings, norms, tied head, and auxiliary parameters

Intended use

This checkpoint is primarily for:

• —continued pretraining
• —research / ablations
• —tracking Veyra training milestones
• —testing tiny model behavior

It is not intended for production use or reliable factual answering.

Known limitations

This model can:

• —hallucinate confidently
• —repeat phrases
• —fail arithmetic
• —fail simple factual questions
• —produce fake code
• —continue in textbook-like or tutorial-like styles

Special Tokens

Important tokenizer special tokens include:

~~~text <|bos|> <|eos|> <|pad|> <|unk|> <|imstart|> <|imend|> <|toolcall|> <|toolresult|> <|context|> <|reasoning|> <|endreasoning|> <|answer|> <|fimprefix|> <|fimmiddle|> <|fimsuffix|> ~~~

Citation / Attribution

If you use or build on this model, please retain attribution to Veyra AI.

License

Apache 2.0.