MyLabs-LLC/Axion-1B-FineWebEdu-200K

Axion 1B FineWeb-Edu — Step 200K

This is an early base-model checkpoint from the Axion three-stage pipeline. It has 1,275,168,768 parameters and was pretrained on FineWeb-Edu with the Axion 16K SentencePiece tokenizer.

Checkpoint

• —Optimizer step: 200,000
• —Training tokens seen: 13,107,200,000
• —Validation cross-entropy: 2.457392
• —Validation perplexity: 11.67432
• —Context length: 32,768 tokens
• —Weight format: BF16 SafeTensors

This checkpoint is not instruction-tuned or chat-tuned. It may generate incorrect, repetitive, biased, or otherwise unreliable text.

Transformers

Custom model code is included because Axion is not a stock Llama architecture. Review the repository code before enabling trust_remote_code.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_NAMESPACE/YOUR_REPOSITORY"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="bfloat16",
    device_map="auto",
)

inputs = tokenizer("The future of science is", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.8)
print(tokenizer.decode(output[0], skip_special_tokens=True))

vLLM

Axion uses vLLM's Transformers modeling backend:

vllm serve YOUR_NAMESPACE/YOUR_REPOSITORY \
  --trust-remote-code \
  --model-impl transformers \
  --dtype bfloat16 \
  --max-model-len 32768

For a 12GB GPU, begin with --max-model-len 8192 to leave sufficient room for the KV cache. A 24GB GPU can use the full context more comfortably.

Upload

From the Axion repository:

hf repos create YOUR_NAMESPACE/YOUR_REPOSITORY --private --exist-ok
hf upload-large-folder YOUR_NAMESPACE/YOUR_REPOSITORY hf_export/macho-1b-finewebedu-20b-step200000

No model license is asserted by this export. The repository owner should select and document a license before making the model public.