pragmaticcs/Triumvirate-Qwopus-MiMo-Ornith-9B-Coder

<div align="center">

     

</div>

Most sub-10B coding models crumble the moment they enter real-world agentic workflows: they either produce clean code but loop endlessly when a shell command fails, or handle tool calls reasonably well while hallucinating obscure API syntax.

Triumvirate is a merge designed to solve that dilemma. It combines three of the most capable specialized fine-tunes of **Qwen 3.5 9B** and fuses their task vectors directly into the base backbone:

• —Algorithmic & Syntax Precision from **Qwopus**
• —SWE-bench Problem Decomposition & Tool Calling from **MiMo-V2.6**
• —Loop-Termination & Error-Recovery Discipline from **Ornith-1.5**

[!IMPORTANT] The result is a lean, blisteringly fast 9B pure-text causal engine with a native 256k context window that runs comfortably on consumer GPUs.

Contents

• —Architectural Specifications
• —Composition & Donor Weighting
• —Merge Methodology & Mathematical Formulation
• —Layer-Stratified Component Policies
• —Agentic Chat Template
• —Recommended Generation Parameters
• —How to Use
• —Citation & References

Architectural Specifications

Composition & Donor Weighting

The foundation checkpoint serves as the structural base (W₀). Three donor models contribute directional task vectors weighted continuously across network depth:

Merge Methodology & Mathematical Formulation

The merge combines TIES-DELLA saliency trimming, consensus sign election, Gated DeltaNet norm stabilization, and continuous sinusoidal depth modulation.

1. Task Vector Formulation

For each donor checkpoint k ∈ {1, 2, 3}, the parameter update delta is isolated relative to the base anchor W₀:

$$ \tauk = Dk - W_0, \quad k \in \{\text{MiMo}, \text{Ornith}, \text{Qwopus}\} $$

2. Asymmetric Sinusoidal Depth Modulation

Task vector mixing coefficients are continuously parameterized over normalized network depth x = l / (L - 1), where l ∈ {0, 1, ..., 31} and L = 32:

$$ u_{\text{qwopus}}(x) = 0.45 \cos^2\left(\frac{\pi}{2} x\right) + 0.25 $$

$$ u_{\text{ornith}}(x) = 0.35 \sin^2\left(\pi x^{0.85}\right) + 0.15 $$

$$ u_{\text{mimo}}(x) = 0.55 \sin^2\left(\frac{\pi}{2} x^{1.20}\right) + 0.20 $$

The donor weights α_k(l) are normalized to form a partition of unity across all layers:

$$ \alphak(l) = \frac{uk(x)}{\sum{j=1}^3 uj(x)}, \quad \sum{k=1}^3 \alphak(l) = 1.0 $$

• —*Lower Layers (x → 0):* Qwopus dominates with α₁(0) ≈ 0.67, ensuring foundational language representations and syntax heads are grounded in Claude 3.5 Opus traces.
• —*Middle Layers (x ≈ 0.5): The sub-linear exponent (x*^0.85) accelerates Ornith's activation to peak across middle transformer blocks with α₂(16) ≈ 0.354, reinforcing state-space continuity and execution discipline.
• —*Top Layers (x → 1): The super-linear exponent (x*^1.20) concentrates MiMo's task vector with α₃(31) ≈ 0.652 into the upper decoders, governing semantic reasoning, multi-turn planning, and final token synthesis.

3. Saliency Trimming (TIES-DELLA Pruning)

To eliminate parameter interference and cross-talk, task vectors are pruned based on parameter energy. Given density parameter ρ = 0.70, an update threshold γ_k is computed per tensor:

$$ \gammak = \text{Quantile}{1 - \rho}\left(\{|\tau_{k, ij}|\}\right) $$

Updates below the top 70% magnitude are zeroed out via a saliency mask:

$$ Mk = \mathbb{I}\left(|\tauk| \ge \gamma_k\right) $$

$$ \hat{\tau}k = \tauk \odot M_k $$

4. Consensus Sign Election & Disjoint Averaging

Surviving task vectors often conflict in directional signs, causing mutual cancellation when averaged naively. A directional consensus sign vector Γ is elected:

$$ \Gamma = \operatorname{sgn}\left(\sum{k=1}^3 \alphak(l) \hat{\tau}_k\right) $$

A binary agreement mask A_k discards parameter updates that oppose the elected consensus sign:

$$ Ak = \mathbb{I}\left(\operatorname{sgn}(\hat{\tau}k) = \Gamma\right) \odot \mathbb{I}\left(\hat{\tau}_k \neq 0\right) $$

The merged task delta is reconstructed using only parameters aligned with the majority direction:

$$ \Delta{\text{TIES}} = \begin{cases} \frac{\sum{k=1}^3 \alphak(l) \left(\hat{\tau}k \odot Ak\right)}{\sum{k=1}^3 \alphak(l) Ak} & \text{if } \sum{k=1}^3 \alphak(l) A_k > 0 \\ 0 & \text{otherwise} \end{cases} $$

The dense layer weights are restored onto the base foundation:

$$ W{\text{dense}} = W0 + \Delta_{\text{TIES}} $$

5. Gated DeltaNet (GDN) Gate Norm Stabilization

In linear attention layers, gate matrices control state retention and output gating via non-linear sigmoid activations. Direct delta merging shifts the operator norm, causing activation saturation or exploding outputs. To guarantee numerical stability, the merged gate weight W_{gate, unscaled} = W₀ + ∑_k α_k τ_k is projected onto the base tensor's Frobenius norm:

$$ W{\text{gate}} = W{\text{gate, unscaled}} \cdot \frac{\|W0\|F}{\|W{\text{gate, unscaled}}\|F} $$

6. Log-Decay and Normalization Parameter Convexity

For state-space logarithmic decay tensors (A_log ∈ (-∞, 0]), biases, and layer normalization parameters, delta blending can violate mathematical boundary constraints. These tensors are merged strictly via convex interpolation:

$$ W{\text{convex}} = \sum{k=1}^3 \alphak(l) Dk $$

Because ∑_k α_k(l) = 1.0, α_k(l) ≥ 0, and D_{k, ij} ≤ 0 for all decay parameters:

$$ \sum{k=1}^3 \alphak(l) D{k, ij} \le \maxk(D{k, ij}) \le 0 \implies \exp\left(W{\text{convex}, ij}\right) \in (0, 1] $$

This guarantees Bounded-Input Bounded-Output (BIBO) stability and prevents exponential divergence in recurrent linear attention states.

Layer-Stratified Component Policies

Agentic Chat Template

This model uses the Improved Chat Template for Qwen 3.x by Olivia Rossi to support multi-tier Chain-of-Thought (CoT) reasoning, dual-format agentic tool execution, automatic error-recovery heuristics, and strict token-waste elimination.

Recommended Generation Parameters

The following parameters are optimal for code synthesis, terminal agent execution, and complex reasoning:

How to Use

Serving via vLLM

vllm serve pragmaticcs/Triumvirate \
  --dtype bfloat16 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-reasoning \
  --reasoning-parser qwen3

Inference via Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pragmaticcs/Triumvirate"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {
        "role": "system",
        "content": "You are a principal software engineer. Think carefully before outputting production-grade code."
    },
    {
        "role": "user",
        "content": "Implement an asynchronous token-bucket rate limiter in Python supporting burst handling and thread-safe redis synchronization."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=2048,
    temperature=0.6,
    top_p=0.95,
    top_k=20,
    do_sample=True,
)

response = tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)
print(response)

Citation & References

• —Qwen/Qwen3.5-9B
• —XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
• —ornith-ai/Ornith-1.5-9B
• —Jackrong/Qwopus3.5-9B-Coder
• —Improved Chat Template for Qwen 3.x

@inproceedings{yadav2023ties,
  title={Resolving Interference When Merging Models},
  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  volume={36},
  pages={7093--7115},
  year={2023}
}

@article{deep2024della,
  title={DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling},
  author={Deep, Pala Tej and Bhardwaj, Rishabh and Poria, Soujanya},
  journal={arXiv preprint arXiv:2406.11617},
  year={2024}
}

@inproceedings{yu2024dare,
  title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
  author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}