eggdog100/Qwen3.6-35B_Zenith

Qwen3.6-35B-A3B — Capability SFT (LoRA, vision-preserved)

A LoRA supervised-fine-tune of Qwen/Qwen3.6-35B-A3B that strengthens math, code, tool-calling, and natural human-like conversation while keeping the model's other abilities — including its vision capability — intact.

Private, self-use derivative. Built entirely on openly-licensed components.

What was changed

• —Method: LoRA SFT on the text decoder only. The vision tower, the 256 routed experts, and the router are frozen — the vision tower ships bit-identical to the base.
• —Trainable: 16.71M params (~0.05% of 35.1B) — LoRA on attention q/k/v/o_proj + the per-layer shared_expert gate/up/down_proj.
• —Base model: Qwen/Qwen3.6-35B-A3B (Apache-2.0), a hybrid linear+full-attention multimodal MoE (40 layers, 256 experts / 8 active, ~3B active params, vision tower).

Training data (full provenance — none distilled from closed frontier models)

All data is openly licensed and either human-authored or generated by open-weights models (DeepSeek-R1, Llama-3.1, Qwen2.5, StarCoder2). No GPT / Claude / Gemini outputs are used anywhere. ~98,074 examples total.

⚠️ NonCommercial note: no_robots and empathetic_dialogues are CC-BY-NC. They are included for the "talks like a human" goal; consequently the resulting weights inherit a non-commercial restriction. (License of this derivative: cc-by-nc-4.0; an NC-clean rebuild is possible by dropping these two sources.)

Training configuration

• —Framework: TRL SFTTrainer + PEFT LoRA, transformers 5.12.
• —LoRA: r=32, α=64, dropout=0.05, bias=none.
• —Optim: bf16, packing (max_len 4096), per-device batch 4 × grad-accum 8 (eff. 32), LR 1e-4 cosine, warmup 3%, 1 epoch, AdamW, grad-clip 1.0.
• —Attention: SDPA (cuDNN attn disabled for head_dim=256); linear-attention layers via flash-linear-attention + tilelang (Hopper).
• —Hardware: 1× NVIDIA H200 NVL (141 GB), ~6–7 h.

Evaluation — vs. independent base measurements (not the inflated official numbers)

Independent reproductions of Qwen3.6-35B-A3B land well below Qwen's official numbers — the official figures use Qwen-specific optimized eval prompts that standard harnesses don't replicate. (A direct re-test of the base on SuperGPQA gives 55.7, vs. the official 64.7.) So Zenith is compared against independent base measurements, not the official numbers:

¹ MMLU-Pro & GPQA-Diamond = Artificial Analysis's own independent evaluation; SuperGPQA = re-tested directly on the base Qwen3.6-35B-A3B. Official Qwen numbers (MMLU-Pro 85.2 / GPQA 86 / SuperGPQA 64.7) are not reproducible with standard harnesses. ² EvalScope, sampled subsets, single-sample (n=1).

Verdict — no meaningful regression. Where independent base numbers exist (MMLU-Pro, SuperGPQA), Zenith equals or slightly beats the base (76.9 vs 75.0; 56.3 vs 55.7). The one outlier — GPQA-Diamond 72 vs AA's 84 — is our small-sample EvalScope run (50 questions, n=1, ~6% answer-extraction failures), not real degradation: the other reasoning axes show no drop.

Conversational style — the primary goal ("talk like a human, not a robot")

The empathy/voice SFT noticeably de-roboticizes the model in conversational & emotional contexts (thinking off):

User: 说说你对"内卷"的真实看法吧，就像朋友聊天那样。 Zenith: 说实话，"内卷"这词儿现在满天飞，真不是年轻人矫情…就像一群人挤在电梯里，明明已经满员了，还在拼命往里挤，结果谁也没上去…消耗的不是体力，是心气儿…人生又不是百米冲刺，对吧？慢慢走，也能到终点。

User: 我今天被领导当众批评了，挺难受的。 Zenith: 听到你这么说，我能感受到你现在一定挺委屈和难过的…先给你一个大大的拥抱吧～…你平时工作已经很用心了，一时的挫折不会掩盖你的闪光点…慢慢来，会好起来的。🌿

GGUF quants (llama.cpp / LM Studio / Ollama)

Converted with `convert_hf_to_gguf.py --no-mtp` — the multi-token-prediction draft head is excluded (it makes the GGUF fail to load otherwise: missing tensor 'blk.40.attn_norm.weight'); normal inference is unaffected. For image input, pass an mmproj file alongside the model.

⚠️ Brand-new architecture caveat: Qwen3.6-35B-A3B (hybrid GatedDeltaNet linear-attention + full-attention MoE) is newly supported in llama.cpp. These GGUFs load and generate correct output, but the runtime kernels for this arch are still maturing upstream, so expect lower throughput than mature architectures until those kernels land. For full-speed serving, use the bf16 weights via vLLM / transformers.

Attribution / licenses

Derivative of Qwen/Qwen3.6-35B-A3B (Apache-2.0; NOTICE retained). Datasets under Apache-2.0 / ODC-BY / CC-BY-NC-4.0 as listed (attribution preserved). Tools: TRL, PEFT, transformers, flash-linear-attention, tilelang, vLLM, EvalScope — all open-source.