Lordnyx/qwen3.5-9b-triple-loop-fase1

Qwen3.5-9B Triple-Loop (Phase 1 — Step 2100)

An experimental fine-tune of Qwen/Qwen3.5-9B using LoopSplit: the middle transformer layers are executed three times in a row, sharing the same weights, and distilled against a stronger teacher model's logits (Qwen/Qwen3.8-27B) with Deep Supervision.

This model scales the LoopSplit architectural paradigm up to 9B parameters with a hybrid Gated-DeltaNet (linear attention) + full-attention backbone, providing the expressive power of a 64-layer virtual depth model while keeping the physical memory footprint of a standard 32-layer 9B model.

What's New in Step 2100 (Cosine LR Decay)

In the initial training session (Step 929), the model used a constant learning rate ($3 \times 10^{-5}$) which led to a training plateau.

In this continuation run (Step 2100):

• —Cosine Learning Rate Decay: The learning rate was decayed smoothly from $1.5 \times 10^{-5}$ down to $1.81 \times 10^{-6}$ over 14,000 seconds of active compute on an NVIDIA B200 GPU.
• —Plateau Broken: Mean KL divergence dropped from $\sim 0.650$ down to $\mathbf{0.30 - 0.36}$.
• —Perfect Mathematics: Achieved 100% accuracy (10/10) on the standard benchmark mathematics suite (up from 70% in the base model and 90% in Step 929), eliminating simple arithmetic skipping errors.
• —Zero Repetition Loops: Fixed all cyclic repetition failure modes observed in the un-decayed checkpoint.

Architecture: LoopSplit

Qwen3.5-9B natively has 32 transformer layers (24 linear-attention / Gated-DeltaNet layers interleaved 3:1 with 8 full-attention layers).

LoopSplit does not add new parameter tensors — it restructures the computation graph:

• —Layers 0–7 (Prefix): Executed once.
• —Layers 8–23 (Middle, 16 layers): Executed three times in a row, reusing the exact same weights across passes ($r0, r1, r_2$).
• —Layers 24–31 (Suffix): Executed once.
• —Effective Virtual Depth: $8 + (16 \times 3) + 8 = \mathbf{64\text{ layers}}$.

The saved checkpoint is restored via wrapper.restore() into a standard 32-layer Qwen3.5-9B. Downstream inference tools and runtimes (llama.cpp, vLLM, transformers) load and run it seamlessly without requiring custom architecture patches.

Benchmark Results

Evaluated on the official 97-item reasoning, logic, coding, and robustness benchmark (greedy decoding, temperature 0.0):

Training Details

• —Teacher Model: Qwen/Qwen3.8-27B (native bf16).
• —Distillation Method: Chunked Top-32 KL divergence with Deep Supervision at intermediate loop boundaries: $$\mathcal{L}{\text{total}} = \mathcal{L}{\text{final}} + 0.15 \times \mathcal{L}{r0} + 0.25 \times \mathcal{L}{r1}$$
• —Data: Curated pool of agentic, logic, coding, and mathematical multi-turn conversations (~48.2k examples, ~28.2M tokens trained).
• —Optimizer: torchao.optim.AdamW8bit with stochastic rounding (bf16_stochastic_round=True).
• —Hardware: NVIDIA B200 GPU on Modal (throughput: ~2,220 tokens/sec).
• —Total Training Progress: 2,100 optimizer steps, 28,172,942 tokens processed.

Files in this Repository

• —model.safetensors + config/tokenizer — Standard Hugging Face format in bfloat16 (~17.9 GB).
• —qwen35-9b-triple-loop-fase1-step2100.gguf — Full precision bfloat16 GGUF (~17.9 GB).
• —qwen35-9b-triple-loop-fase1-step2100-Q5_K_M.gguf — Quantized GGUF (Q5_K_M, ~6.15 GB).

How to Run with llama.cpp

The GGUF weights use llama.cpp's native qwen35 hybrid architecture (DeltaNet + SDPA attention):

# Run server with CUDA offload:
llama-server -m qwen35-9b-triple-loop-fase1-step2100-Q5_K_M.gguf -ngl 99 -c 4096 --port 8080

# Or CLI inference:
llama-cli -m qwen35-9b-triple-loop-fase1-step2100-Q5_K_M.gguf -ngl 99 -c 4096 -p "Solve the following problem step by step: What is the sum of angles in a regular decagon?"

Prompt Format (Chat Template)

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
<think>
{reasoning}
</think>
{final_answer}