aws882/Qwen3.5-9B-UD-Q2_K_XL-stages

Qwen3.5-9B-UD-Q2KXL pre-split for DPI

Pre-split GGUF weights of `unsloth/Qwen3.5-9B-GGUF` (Q2KXL Unsloth Dynamic quant, 3.93 GB source) into per-stage GGUFs ready for distributed pipeline inference (DPI). Each stage GGUF is a self-contained GGUF that vanilla llama.cpp can load standalone — the splitter keeps token_embd.weight on every stage so middle stages pass libllama's load-time supported-tensor check.

Layout

Source model: 32 transformer layers (qwen35 arch). Auto-split evenly; the last stage carries output_norm + lm_head so it's slightly larger than the rest.

stages-2/

stage0  layers [0, 16)  → 1805 MB, 214 tensors
stage1  layers [16, 32) → 2682 MB, 215 tensors  (carries lm_head)

stages-4/

stage0  layers [0,  8)  → 1181 MB, 108 tensors
stage1  layers [8,  16) → 1179 MB, 108 tensors  (middle, vanilla-libllama loadable)
stage2  layers [16, 24) → 1214 MB, 108 tensors  (middle, vanilla-libllama loadable)
stage3  layers [24, 32) → 2024 MB, 109 tensors  (carries lm_head)

How the splits were made

gguf_stage_split \
    --input Qwen3.5-9B-UD-Q2_K_XL.gguf \
    --output-dir stages-N \
    --auto-split N \
    --prefix qwen3.5-9b-q2kxl

The splitter is part of the obit DPI tooling (cc-headless-v2/src/bin/gguf_stage_split.rs). Each stage GGUF has its layer indices renumbered starting at 0 (blk.0, blk.1, …) so that llama.cpp's existing GGUF model loader works without modification on each stage.

Loading on llama.cpp / libllama

Every stage in this repo is loadable by stock llama.cpp (tested at b8937). Middle stages keep token_embd.weight purely so the loader's supported-tensor check passes; the embedding is not used at inference time on those stages.

License

Same as the upstream unsloth/Qwen3.5-9B-GGUF model.