Cadododoom/Qwen3.6-35B-A3B-DSV4Pro-FP4

Qwen3.6-35B-A3B-DSV4Pro-FP4

Model Summary

This is the NVFP4 (W4A16) quantized version of the Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill model.

Key Characteristics:

• —Vision Tower Kept: Both the speculative decoding MTP heads have been pruned to reduce model overhead, but the visual tower (`model.visual`) is fully preserved. To guarantee multimodal capability and high accuracy, the visual tower is kept unquantized in high-precision bfloat16.
• —NVIDIA ModelOpt FP4 (GS16) Quantization: Quantized using NVIDIA ModelOpt with Group Size 16 (GS16).
• —Note on Group Size: While we tested Group Size 128 (GS128) to shrink the footprint further, the vLLM Marlin FP4 CUDA kernel (marlin_mm) only supports a group size of 16. Attempts to serve GS128 will result in a serving engine crash (Invalid thread config). Therefore, GS16 is the only viable serving configuration.
• —Disk Footprint: ~21 GB.

Evaluation Benchmarks (GS16, thinking-on)

Evaluated using the local test suite (temperature 0.6, top_p 0.95):

• —GSM8K Accuracy: 85.00% (17/20 correct)
• —ARC-Challenge Accuracy: 80.00% (16/20 correct)
• —GPU KV Cache (RTX 5060 Ti / vLLM): 534,588 tokens (FP8 KV Cache enabled). At a target concurrency of 32, this allocates a physical slice of 16,705 tokens per concurrent request (prefix caching enabled).

Original Distill Model Information (nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill)

On Qwen3.6-35B-A3B (MoE, 3B active), we use LoRA to *distill the way DeepSeek-V4-Pro reasons (with thinking-on) plus its agentic behavior — purpose-built as a fast task orchestrator (decompose → delegate → verify) for Lynn Agent. This is the MoE counterpart of the 27B Dense sister model: same R6000 GPU, same teacher, same recipe, on a sparse architecture. ⚠️ Distilling a thinking style ≠ distilling knowledge/capability: the goal is "learn how to reason and how to converge*", not to inject knowledge or raise the capability ceiling.

Training details

• —Base: Qwen3.6-35B-A3B (MoE, 3B active, BF16 base)
• —Method: LoRA, r = 64, α = 128, dropout = 0.05, targets = all attention + MLP projections
• —Optim: pagedadamw8bit, cosine LR, warmup 0.03, ~1 epoch
• —Teacher: DeepSeek-V4-Pro (thinking-on + agentic)
• —Data: ~1,842 distillation samples (lynn_prod spec). Trajectories = DS-V4-Pro multi-step reasoning under thinking-on (<think>) + ReAct-style tool calls (think one step → call one tool → observe → loop).
• —The tool "execution results" are SIMULATED, not actually run — each result line is improvised by a small fast model (DeepSeek-V4-Flash) role-playing the runtime, not real sandbox execution.
• —Training masks those fabricated results — the model learns only "how to think / how to call tools", never the made-up outputs (so it doesn't learn to fabricate tool returns).

Attribution (the method is not original — it is a combination of published techniques)

• —ReAct (interleaved reasoning + acting): Yao et al., 2022, arXiv:2210.03629 (ICLR 2023)
• —STaR (bootstrapping reasoning traces): Zelikman et al., 2022, arXiv:2203.14465
• —Self-Instruct / Baize self-chat: Wang et al., 2022; Xu et al., 2023, arXiv:2304.01196
• —AgentTuning: Zeng et al., 2023, arXiv:2310.12823
• —ToolBench / ToolLLM: Qin et al., 2023, arXiv:2307.16789
• —DeepSeek-R1 reasoning distillation: DeepSeek-AI, 2025, arXiv:2501.12948

Serving & Inference with vLLM

Load using the standard compressed-tensors path inside vLLM. Here is the recommended serving configuration:

vllm serve /path/to/Qwen3.6-35B-A3B-DSV4Pro-Vision-FP4-GS16 \
  --served-model-name nvidia/Agents-A1-FP4 \
  --enforce-eager \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --quantization compressed-tensors \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --max-model-len 128000 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching

🇨🇳 中文版 (Chinese Version)

模型摘要

这是 Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill 模型的 NVFP4 (W4A16) 量化版本。

• —保留视觉塔 (Visual Tower): 我们移除了投机解码的 MTP 头以降低运行开销，但完整保留了视觉塔 (`model.visual`)。视觉塔保持未量化的 bfloat16 高精度格式，以确信优秀的多模态理解能力。
• —NVIDIA ModelOpt FP4 (GS16) 量化: 使用 NVIDIA ModelOpt 以 Group Size 16 (GS16) 进行 PTQ 校准。
• —关于组大小的说明: 虽然我们测试了 Group Size 128 (GS128) 以进一步减小模型体积，但在 vLLM 中，Marlin FP4 CUDA 算子 (marlin_mm) 目前仅支持 group size 为 16。尝试加载 GS128 模形将导致服务崩溃（提示 Invalid thread config）。因此，GS16 是唯一可用的运行服务配置。
• —磁盘大小: ~21 GB。