UniversalComputingResearch/Atom3.4m

[image]

Atom 3.4m

Atom is a 3.4M parameter causal language model developed by Universal Computing Research. It was pretrained from scratch as a compact research model for studying language-model architecture, data curricula, and small-model benchmarking.

Model details

• —Architecture: causal decoder-only language model
• —Parameters: 3,412,800
• —Layers: 7
• —Hidden size: 192
• —Attention: 3 query heads and 1 key-value head (grouped-query attention)
• —Head dimension: 64
• —Feed-forward size: 480
• —Context length: 512 tokens
• —Positional encoding: rotary position embeddings (RoPE)
• —RoPE Theta = 5000.0
• —Normalization: RMSNorm
• —Activation: gated SiLU feed-forward network
• —Vocabulary size: 4,096 tokens
• —Tokenizer: custom byte-level BPE, exposed as GPT2TokenizerFast
• —Training tokens: approximately 5 billion
• —License: Apache-2.0

The model uses tied input and output embeddings. Its custom attention implementation combines grouped-query attention with XSE.

Tokenizer

Atom uses a custom byte-level BPE tokenizer trained specifically for this pretraining corpus. The tokenizer has a vocabulary of 4,096 tokens and includes dedicated padding, beginning-of-sequence, end-of-sequence, unknown, and end-of-text tokens.

Training data and curriculum

Atom was trained on a curriculum combining general web text, educational material, synthetic textbook-style content, and mathematical data. The mixture changed gradually during training: general web data was emphasized earlier, while educational, synthetic, and mathematical material received more weight later.

Approximate proportions over the complete training run were:

These percentages describe the approximate aggregate sampling mixture rather than exact document counts. Refer to the individual dataset cards for their source information, licenses, and usage conditions.

Intended use

This is a small base language model intended for research and benchmarking. It may be useful for experiments involving compact architectures, pretraining curricula, tokenization, evaluation pipelines, and resource-constrained inference.

Atom is a base model and has not been instruction-tuned or aligned for assistant-style interaction.

Evaluation

Atom was evaluated with EleutherAI's lm-evaluation-harness.

lm-evaluation-harness

Limitations

Atom is a very small model and should not be expected to produce reliable factual, safety-critical, or instruction-following outputs. Its short context window and limited capacity constrain coherence, knowledge recall, reasoning, and long-form generation.

The model may reproduce errors, biases, or undesirable patterns present in its training data. It has not undergone dedicated safety training and should not be used for high-stakes decisions.