PrometheanStudio/talos-mini-1m-oasst1

Talos Mini 1M — OASST1

Experimental 1,000,320-parameter Talos Mini language model trained on a small subset of OpenAssistant/oasst1.

This repository contains the native Talos checkpoint and native Talos Byte-Level BPE tokenizer.

Model specifications

• —Parameters: 1,000,320
• —Architecture: TalosGPT decoder-only transformer
• —Hidden size: 128
• —Layers: 3
• —Attention heads: 8
• —KV heads: 4
• —Head dimension: 16
• —Dense FFN intermediate size: 512
• —Vocabulary size: 1,024
• —Maximum sequence length: 512
• —Tokenizer merges: 764
• —Training data: 1,800 OASST1 messages
• —Evaluation data: 200 held-out OASST1 messages
• —Training sequence length: 64
• —Training batch size: 4
• —Optimizer: AdamW
• —Learning rate: 3e-3

Training

The model was trained using the existing Talos training stack.

Published checkpoint:

step-26925.pt

The checkpoint contains the native Talos model state and training metadata.

Results

At step 26,925:

• —Validation loss: approximately 2.254
• —Validation perplexity: approximately 9.52
• —Validation next-token accuracy: approximately 35%
• —Evaluated tokens: 102,816

These results come from a small research experiment and are not a benchmark of general language-model capability.

The model can generate recognizable language patterns, but it is not a coherent general-purpose conversational model.

Tokenizer

The tokenizer is the native Talos ByteLevelBPETokenizer.

It is not a Hugging Face GPT-2 tokenizer conversion.

The tokenizer uses Talos-native byte IDs and merge IDs.

• —Vocabulary: 1,024
• —Merges: 764

Status

Demonstrated

• —1M-parameter Talos model construction
• —Native Talos tokenizer
• —OASST1 data loading
• —Train/evaluation split
• —Training
• —Checkpoint saving/loading
• —Held-out evaluation
• —Text generation
• —KV-cache equivalence testing
• —Greedy generation

Limitations

This is an experimental dev-scale model trained on a small dataset and short sequence length.

It is not intended to compete with modern general-purpose language models.

The 1M model is a scaling experiment between the 254K Talos Mini prototype and larger future Talos configurations.

Long-term Talos scaling targets are research goals and are not represented here as demonstrated capabilities.