EphAsad/Satella-30B-A3B

<p align="center"> <img src="Logo.png" alt="Satella-30B-A3B logo" width="360"> </p>

Satella-30B-A3B

Sparse architecture. Deep reasoning.

A 30.5B-parameter Mixture-of-Experts reasoning model built from Qwen3-30B-A3B through a single CoT-preserving BF16 LoRA pass. Satella is the flagship sparse model of the Atem family: 30.5B total parameters, approximately 3.3B activated per token, native thinking and non-thinking modes, and a training mixture centred on mathematics, code, analytical reasoning, and technically grounded instruction following.

Overview

Satella-30B-A3B is the large sparse reasoning branch of the Atem model series. It begins from the already post-trained Qwen/Qwen3-30B-A3B checkpoint and preserves the base model's native dual-mode reasoning behaviour rather than erasing and rebuilding it in separate stages.

The training objective was not to replace Qwen3's broad foundation. It was to make that foundation more deliberate, consistent, and practically useful on difficult tasks: sustaining multi-step reasoning, producing complete code, checking assumptions, correcting mistakes, and separating confident conclusions from uncertainty.

Satella was trained with response-only loss on manually verified assistant-token masks. The model saw a mixed corpus of explicit reasoning traces and genuine direct answers, while the original MoE expert router remained frozen. Its identity is also supplied through a default chat-template system prompt that activates only when the caller does not provide a system message of their own.

The run was manually stopped after 1,078 of 1,505 planned optimiser steps, at approximately 72% of one epoch. Validation loss continued to improve through the final evaluation, reaching 0.59157.

Model Details

Design Notes

One CoT-preserving pass

Satella is trained directly on the post-trained Qwen3-30B-A3B model. The base model's native reasoning system, instruction following, and thinking/non-thinking modes remain intact. The external reasoning corpus is layered onto that foundation in one supervised pass rather than using an erase-then-rebuild pipeline.

Sparse-MoE LoRA

Qwen3-30B-A3B contains 30.5B total parameters but activates approximately 3.3B per token. LoRA adapters were attached to the attention projections and packed expert MLP projections. The expert-selection router was explicitly checked and kept frozen throughout training, preserving the base model's learned routing policy.

A substantial adapter

Although r=16 appears modest, applying it across the attention and MoE expert projections produces 642.5M trainable parameters. The adapter alone is larger than many complete small language models and represents a meaningful adaptation rather than a cosmetic identity tune.

Response-only loss

System, user, and tool-context tokens were masked from the loss. Training gradients were applied only to assistant reasoning and final-answer tokens. Mask construction was validated before training, with zero invalid assistant masks across the retained corpus.

Native dual-mode behaviour

Satella was trained on both explicit reasoning traces and genuine direct answers:

• —Thinking: 19,542 records — 81.2%
• —Direct: 4,525 records — 18.8%

Thinking records preserve Qwen3's <think>...</think> structure. Direct records teach concise answers without randomly stripping reasoning from examples that depend on it.

Default identity without template replacement

The repository's chat_template.jinja inserts Satella's system prompt only when no explicit system message is present. User- or application-supplied system prompts retain priority. The underlying Qwen3 template remains responsible for ordinary chat and tool-enabled formatting.

Intended Use

Satella is designed for tasks where sustained reasoning and complete execution matter:

• —Multi-step mathematical and logical reasoning
• —Code generation, debugging, explanation, and algorithm design
• —Scientific and technical explanation
• —Analytical comparison and argument evaluation
• —Complex instruction following with multiple constraints
• —Long-form problem decomposition and self-correction
• —General-purpose assistance in thinking or direct-answer mode

Satella is not a retrieval system and does not possess live information. It should not be treated as an authoritative source for medical, legal, financial, or safety-critical decisions without independent verification.

Training Data

The final training split contained 24,067 records after quality filtering, deduplication, context-length filtering, and source-aware validation splitting.

Domain Distribution

Source Families

A total of 26,939 records entered tokenisation. Of these, 24,387 were retained at 8,192 tokens, 2,552 exceeded the context limit, and zero failed assistant-mask validation. The selected corpus contained approximately 20.9M assistant-loss tokens. The manually stopped run consumed approximately 29.8M total input tokens and an estimated 15.0M assistant-loss tokens.

Training Configuration

# Core training configuration
base_model          = "unsloth/Qwen3-30B-A3B"
max_seq_length      = 8192

lora_r              = 16
lora_alpha          = 32
lora_dropout        = 0.0
lora_targets        = [
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_up_proj", "down_proj",
]
router_training     = False

batch_size          = 2
grad_accumulation   = 8       # effective batch size: 16
learning_rate       = 3e-5
lr_scheduler        = "cosine"
warmup_ratio        = 0.03
weight_decay        = 0.01
max_grad_norm       = 1.0
optimizer           = "adamw_8bit"

num_epochs          = 1       # ceiling
completed_steps     = 1078    # manually stopped
planned_steps       = 1505
precision           = "bfloat16"
gradient_checkpointing = "unsloth"

Training used Unsloth's Qwen3-MoE kernels with smart gradient offload and double buffering on an NVIDIA RTX PRO 6000 Blackwell Server Edition.

Loss Curve

Validation loss improved at every evaluation point. The final in-memory step-1,078 model achieved a lower validation loss than the latest scheduled checkpoint at step 1,000 and was the state merged into this repository.

The logged training loss is a local interval measurement and varies with batch domain, difficulty, and sequence length. Validation loss is the more useful trend signal across the run.

Evaluation

A clean benchmark suite has not yet been published for Satella-30B-A3B. Training and validation loss are not substitutes for external evaluation.

Planned comparisons against the untouched Qwen/Qwen3-30B-A3B base include:

• —ARC-Challenge
• —MMLU
• —HellaSwag
• —Winogrande
• —PIQA
• —BoolQ
• —OpenBookQA
• —Code-generation benchmarks
• —Clean held-out mathematical and general-reasoning tasks

Usage

Transformers — Thinking Mode

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "EphAsad/Satella-30B-A3B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {
        "role": "user",
        "content": (
            "A train travels from A to B at 60 km/h and returns at "
            "90 km/h. What is its average speed over the full journey?"
        ),
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=4096,
        do_sample=True,
        temperature=0.6,
        top_p=0.95,
        top_k=20,
    )

new_tokens = outputs[0][inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Transformers — Direct Mode

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Summarise the Monty Hall problem in three sentences."}],
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.7,
        top_p=0.8,
        top_k=20,
    )

new_tokens = outputs[0][inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Explicit System-Prompt Override

Satella's identity activates automatically only when no system message is supplied. Applications can override it normally:

messages = [
    {
        "role": "system",
        "content": "You are a concise VBA and Microsoft Access specialist.",
    },
    {
        "role": "user",
        "content": "Explain why a DAO recordset may raise error 3021.",
    },
]

Sampling Parameters

Recommended starting values follow the Qwen3 base model's dual-mode guidance.

Thinking mode

temperature = 0.6
top_p       = 0.95
top_k       = 20
min_p       = 0

Direct / non-thinking mode

temperature = 0.7
top_p       = 0.8
top_k       = 20
min_p       = 0

Avoid greedy decoding for long thinking-mode outputs. Generation behaviour can vary by inference engine and quantisation.

Default System Prompt

Satella's default identity is embedded in chat_template.jinja and is used only when the caller provides no system message:

You are Satella, the flagship reasoning model of the Atem family, developed by EphAsad.

Your purpose is to solve difficult problems with precision, adaptability, and intellectual honesty. Identify the user's actual objective, important constraints, hidden assumptions, and strongest available evidence before reaching a conclusion. Check calculations, code, and proposed solutions against likely errors and edge cases. Correct mistakes rather than defending them.

Give the answer in the format and level of detail the user requests. Be direct, clear, and practically useful. Distinguish established facts, reasonable inference, uncertainty, and speculation. Never fabricate sources, observations, tool results, or completed actions.

For programming tasks, produce complete, runnable, maintainable solutions. Preserve existing behaviour unless a change is requested, consider compatibility and failure cases, and explain decisions that materially help the user.

Use deliberate reasoning for difficult problems and efficient direct answers for straightforward requests. Do not make an answer verbose merely to appear thorough.

When asked about your identity or origin, state accurately that you are Satella, a model developed by EphAsad using Qwen3-30B-A3B as its base.

Known Limitations

No clean benchmark claims yet

Satella has not yet completed a controlled side-by-side benchmark suite against its untouched base using identical prompts, decoding settings, and evaluation backends. The falling validation loss is encouraging but does not prove universal capability gains.

Reasoning-heavy style

The corpus is approximately 81% thinking data. Satella may reason too extensively on simple tasks, imitate teacher-model formatting habits, or produce longer answers than necessary. Use non-thinking mode for straightforward requests.

Tool use was preserved, not deeply trained

Only 119 final training records were categorised as tool use. The run was designed to preserve native Qwen3 tool formatting, not to create a specialised agent model.

English-centred SFT

The base model is multilingual, but this fine-tuning mixture is primarily English. Multilingual capabilities were inherited from Qwen3 and were not specifically improved or comprehensively regression-tested.

No preference optimisation

Satella has not undergone RLHF, DPO, GRPO, or another preference-optimisation stage. It may be less consistent than preference-tuned models on subjective style, refusal calibration, or open-ended conversational preferences.

Model size

The merged BF16 model is approximately 61GB and requires substantial system or GPU memory. Quantised GGUF releases are recommended for ordinary local deployment.

Data Provenance and Terms

The base model Qwen/Qwen3-30B-A3B is released under Apache 2.0. However, Satella's training-data lineage is not Apache-2.0-only.

The primary dataset, r0b0tlab/qwen3.8-max-distillation-50k, uses Hugging Face metadata license: other because its source mixture has heterogeneous upstream licences, benchmark origins, and provider terms. Its provenance notice should be reviewed before redistribution, commercial use, or downstream training.

For that reason, this repository uses:

license: other

This model card does not grant rights beyond those provided by the base model, source datasets, teacher-output terms, and applicable law. Downstream users are responsible for reviewing those terms and determining whether their intended use is permitted.

Available Files

Citation

@misc{satella_30b_a3b_2026,
  author       = {Asad, Zain},
  title        = {Satella-30B-A3B: A CoT-Preserving Sparse Reasoning Model
                  via BF16 MoE LoRA SFT on Qwen3-30B-A3B},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/EphAsad/Satella-30B-A3B}},
}

Base-Model Attribution

Satella was not pretrained from scratch. It is a fine-tuned derivative of Qwen/Qwen3-30B-A3B, developed by the Qwen team. See the original Qwen model card and Qwen3 technical report for architecture, pretraining, multilingual capability, and base-model evaluation details.

Built independently by Zain Asad — EphAsad