CoolFace
Modelpublic

AEON-7/AEON-DFlash-Qwen3.6-35B-A3B

sourceHugging Facemitupdated 3mo agoView on Hugging Face
3likes446downloads
Model Card

AEON DFlash — Qwen3.6 / Ornith 35B-A3B

[image]

A DFlash speculative-decoding drafter for `Qwen/Qwen3.6-35B-A3B` and its derivatives — including `Ornith-1.0-35B-AEON-Ultimate-Uncensored` (a light RL post-train of that base). ~0.9 GB. Gives ~1.9× single-stream decode on a DGX Spark with no quality loss (lossless speculative decoding: the target model verifies every token).

What makes it different

This drafter is 8 layers of full attention — deliberately no sliding-window attention (SWA). That single design choice is the difference:

This drafter (AEON)Typical SWA drafter (e.g. z-lab)
Layers8 × full attention6 × (5 sliding + 1 full), window 4096
KV page sizealigns with the target's pageslarger (windowed) → mismatch
Serves on the AEON vLLM imageyes, no patchneeds a kv_cache_utils patch

Why the no-patch part matters. Qwen3.6-35B-A3B / Ornith is a hybrid stack — full attention + GatedDeltaNet (Mamba) recurrent state. vLLM unifies KV-cache page sizes across layer types by scaling block_size, but Mamba/GDN state pages are shape-based and don't scale. A drafter with SWA layers introduces a larger KV page that can't be reconciled with the GDN pages → vLLM aborts at engine init with unify_kv_cache_spec_page_size … AssertionError (see issue #1). An all-full-attention drafter has no oversized page, so it loads cleanly on the stock AEON image — a true drop-in.

It also targets the right hidden states for this stack: DFlash reads target layers [1, 10, 19, 28, 37] of the 40-layer 35B-A3B model.

A note on KV cache (and why SWA doesn't help here)

This model serves with BF16 KV cache, and a sliding-window drafter does not change that. The BF16-KV requirement comes from two places, neither of which is the drafter's window type:

  1. 1.the vision tower (non-causal vision attention → FA2 backend only → no FP8 KV), and
  2. 2.DFlash's non-causal verification (block-parallel → FA2/FLEX only → no FP8 KV).

KV dtype is a property of the main model's attention backend, not the drafter's layers. SWA changes the drafter's own (tiny, windowed) KV footprint only — it can't unlock FP8 KV for the target.

Usage

Point any Qwen3.6-35B-A3B / Ornith deploy's --speculative-config at this drafter:

bash
  --speculative-config '{"method":"dflash","model":"/drafter","num_speculative_tokens":6}'

Validated on a DGX Spark (GB10) with the Ornith NVFP4 QuickStart. ⚠️ Keep --max-num-seqs ≤ 16 with DFlash on the Spark (unified-memory limit — see the QuickStart).

Performance

Head-to-head on Ornith-1.0-35B NVFP4 (DGX Spark GB10 · n=6 · max-num-seqs 16 · single stream):

Drafterc=1 decodeAccept lengthDraft accept ratePatch needed
AEON (8L full-attn) — this repo72.9 tok/s3.7145.1%none
z-lab (6L SWA)74.4 tok/s3.3539.2%KV page-size hotfix

Speed is effectively tied (c=1 within ~2% — noise; this drafter is slightly ahead at c=8 aggregate). This drafter wins on acceptance — its 8 full-attention layers draft more accurately (3.71 vs 3.35 tokens accepted per step; 45% vs 39%) than the SWA variant's 6 layers, which trade accuracy for a cheaper forward pass. Combined with no patch required and no padded-Mamba-page overhead, it's the recommended default. Both are lossless (the target verifies every token); ~1.9× over plain decode either way.

Credits

  • —Drafter: AEON-7. Method: DFlash speculative decoding.
  • —KV page-size diagnosis + the patch for SWA-drafter compatibility: @newjordan (issue #1 / PR #2).

License: MIT.