AlreadyAI/Bala-30B-A3B

Bala v1 (30B-A3B)

Bala v1 is a coding-and-reasoning model developed by TushLab, the model lab of AlreadyAI. It is produced by attention-only QLoRA fine-tuning of Qwen3-30B-A3B-Thinking-2507 (MoE, ~30B total / ~3B active parameters) on a small mixture of open reasoning-distillation corpora. It is an honest proof-of-capability release: a real, decontaminated improvement in mathematics over its base, with no claim of a coding gain and no frontier claim.

• —Repository: AlreadyAI/Bala-30B-A3B · Release: v1.0
• —Developed by: TushLab (AlreadyAI)

Transparency: Bala is a fine-tune of the Apache-2.0 model Qwen/Qwen3-30B-A3B-Thinking-2507. It is trained to self-identify as "Bala, made by TushLab"; the underlying base is disclosed here and in the accompanying report.

Files & usage

This repo ships both the fully-merged model (at the repo root — load directly) and the standalone LoRA adapters (adapters/). adapters/checkpoint-2556 is the released epoch-3 winner; checkpoint-1700 is the epoch-2 checkpoint. The merged weights and (adapter + base) are equivalent.

Direct (merged weights):

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("AlreadyAI/Bala-30B-A3B")
model = AutoModelForCausalLM.from_pretrained("AlreadyAI/Bala-30B-A3B",
                                             torch_dtype="auto", device_map="auto")

Adapter on the base (equivalent):

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3-30B-A3B-Thinking-2507"
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "AlreadyAI/Bala-30B-A3B",
                                  subfolder="adapters/checkpoint-2556")

What it is / isn't

• —Is: a math-reasoning booster. On MATH-500, pass@1 improves 53.6 → 77.2 (+23.6) over the base.
• —Isn't: a coding improvement (HumanEval+ −3.7, MBPP+ +3.4, GSM8K flat near ceiling), and not competitive with closed frontier models. See Evaluation.

Evaluation (greedy pass@1, full sets)

Base = unmodified Qwen3-30B-A3B-Thinking-2507, evaluated identically. Full methodology, the 4-config hyperparameter comparison, and eval logs are in the technical report and results/.

Decontamination

We ran n-gram containment + exact/substring matching between the entire data pool and all four benchmarks. The data this model was trained on (offline corpora + identity) is clean against all four benchmarks at n=13 and the stricter n=8. (An auxiliary pool of self-generated traces, not used to train this model, contained ~12% MBPP+ overlap; it is disclosed in the paper and is not part of this model's training data.) Report: results/decontamination/report.json. Method: n-gram/exact only — does not catch deep paraphrase.

Training

• —Method: QLoRA, 4-bit NF4 base / bf16 compute, LoRA on q,k,v,o_proj only (MoE expert MLPs frozen). Config B: rank 128, α 256, LR 2.5e-4, cosine, warmup 0.03, 3 epochs, seq-len 4096, effective batch 16, seed 0.
• —Data: 15,296 examples — OpenCodeReasoning (3k), Mixture-of-Thoughts code/math/science (3k each), OpenThoughts3 (3k), identity (296). Offline traces treated as verified-by-construction (not re-executed).
• —Hardware: 1× AWS g6e.12xlarge (4× L40S 48 GB, no NVLink), single-GPU per config.

Intended use & limitations

Research and experimentation on math/reasoning tasks. Single-seed results (no variance reported); coding is not improved over base; identity data is mixed in (model presents as "Bala"). Not for high-stakes use without independent evaluation. Inherits the base model's licenses, biases, and context behavior.

License

Apache-2.0 (base and all released artifacts).

Paper

Technical report: "Distilling Open Reasoning Corpora into a 3B-Active MoE on Commodity GPUs: What Transfers, What Doesn't, and a Decontamination Check" — published on Zenodo, DOI 10.5281/zenodo.22552056 (Apache-2.0 / CC BY 4.0). Full text also in paper/.

Citation

@misc{chauhan2026bala,
  title  = {Distilling Open Reasoning Corpora into a 3B-Active MoE on Commodity GPUs: What Transfers, What Doesn't, and a Decontamination Check},
  author = {Chauhan, Tushar},
  year   = {2026},
  publisher = {Zenodo},
  doi    = {10.5281/zenodo.22552056},
  url    = {https://doi.org/10.5281/zenodo.22552056}
}