pyros-vault/Ornith-1.5-35B-A3B-fixed-mtp-NInfer

Ornith-1.5-35B-A3B fixed-MTP for NInfer-4090

This repository contains a single-file, mixed-precision .ninfer conversion of shisa-ai/Ornith-1.5-35B-A3B-MTP, built and tested with UDPSendToFailed/ninfer-4090 on a 24 GB RTX 4090 (sm_89).

The artifact contains the Ornith target model, Shisa AI's replacement MTP head, the Vision tower, NInfer's optimized proposal head, and embedded Qwen3.6 DFlash draft weights. It is not a Safetensors or GGUF checkpoint and is not intended for Transformers, llama.cpp, or unrelated NInfer forks.

Why this upstream matters: this is the Ornith variant whose MTP component was deliberately repaired. Shisa AI replaced the native head that had underperformed with a Qwen3.6-initialized, KL-distilled head intended for working speculative decoding. This conversion retains it, and the local MTP4 test below confirmed 66.94% accepted draft tokens with zero fallback steps.

Quick facts

Model provenance

The direct source keeps the target transformer, embeddings, LM head, tokenizer, and multimodal components from ornith-ai/Ornith-1.5-35B-A3B. Its native MTP head was replaced with a Qwen3.6-initialized head that Shisa AI reports was trained through full-vocabulary KL distillation after the native head underperformed. The replacement is published separately as model-mtp.safetensors and is code-heavy by design. See the source model card for training details and upstream attribution.

This NInfer conversion preserves that MTP component and additionally embeds the Apache-2.0 DFlash checkpoint from Z-Lab. DFlash targets the shared Qwen3.6 architecture rather than this exact Ornith tune. NInfer verifies every proposal with the target model; the pairing worked strongly in the bounded test below, but acceptance remains prompt-dependent.

Frontend compatibility note

NInfer validates tokenizer, processor, and chat-template resources against its registered Qwen3.6 frontend. Five source frontend resources already matched exactly. The source chat_template.jinja had SHA-256 182e77dd83bd8e9ca818b240b82e28f243762cd5dda32e6eef327df7b1cd107e, while NInfer requires e84f32a23fdda27689f868aa4a1a5621f41133e51a48d7f3efcbea2839574259.

The original template was preserved before conversion; the artifact embeds the exact NInfer-compatible Qwen3.6 template. Model weights, tokenizer data, and processor resources were otherwise retained. The Qwen3.6 product name coexisting with Qwen3_5MoeForConditionalGeneration / qwen3_5_moe identifiers is expected.

What is inside the bundle

The converter validated 1,045 BF16 target tensors across 17 source files (16 main shards plus the replacement MTP shard) and 69 BF16 DFlash tensors before writing 934 runtime tensors plus six embedded frontend resources.

Q4G64_F16S, for example, means 4-bit weights quantized in groups of 64 with FP16 scales; Q5/Q6 use the same group size, while W8G32_F16S uses 8-bit weights in groups of 32.

NInfer loads only the components selected at startup:

KV state, workspaces, CUDA Graphs, and concurrency add further runtime memory. The published conversion report is path-sanitized and contains the full tensor-format and environment summary.

Download

Install the Hugging Face CLI, then download the complete repository:

hf download pyros-vault/Ornith-1.5-35B-A3B-fixed-mtp-NInfer `
  --local-dir .\Ornith-1.5-35B-A3B-fixed-mtp-NInfer

Verify the model artifact:

Get-FileHash `
  '.\Ornith-1.5-35B-A3B-fixed-mtp-NInfer\Ornith-1.5-35B-A3B-MTP.ninfer' `
  -Algorithm SHA256

Serving with the intended NInfer runtime

Build a current native sm_89 version of UDPSendToFailed/ninfer-4090. Older release binaries or other forks that predate the qwen3_6_35b_a3b-v2 target may not load this artifact.

The examples use the conservative 4,096-token profile tested on one 24 GB RTX 4090.

Text with MTP4

.\ninfer-serve.exe '.\Ornith-1.5-35B-A3B-MTP.ninfer' `
  --kv-dtype rk4v4-e8 `
  --spec mtp --draft-tokens 4 --lm-head-draft `
  --max-context 4096 --kv-capacity 4096 --prefill-chunk 512 `
  --preserve-thinking --no-cuda-graph

Text with DFlash7

.\ninfer-serve.exe '.\Ornith-1.5-35B-A3B-MTP.ninfer' `
  --kv-dtype rk4v4-e8 `
  --spec dflash --draft-tokens 7 --lm-head-draft `
  --max-context 4096 --kv-capacity 4096 --prefill-chunk 512 `
  --preserve-thinking --no-cuda-graph

Vision with MTP4

.\ninfer-serve.exe '.\Ornith-1.5-35B-A3B-MTP.ninfer' `
  --vision --vision-max-tokens 1024 `
  --kv-dtype rk4v4-e8 `
  --spec mtp --draft-tokens 4 --lm-head-draft `
  --max-context 4096 --kv-capacity 4096 --prefill-chunk 512 `
  --preserve-thinking --no-cuda-graph

The server exposes OpenAI-compatible /v1/chat/completions and /v1/responses plus Anthropic-compatible /v1/messages. Compatible-prefix reuse is enabled by default, and current builds report it as usage.prompt_tokens_details.cached_tokens.

Local verification results

These are bounded single-run smoke/integration measurements, not a benchmark suite. Text rows used the same code-generation prompt with greedy decoding, thinking disabled, rk4v4-e8 KV, a 4,096-token KV capacity, 512-token prefill chunks, and CUDA Graphs disabled.

The Vision row uses a different, very short multimodal fixture and is not directly comparable to the text rows. Its 428-token expanded prompt used --vision-max-tokens 1024; preprocessing plus Vision execution took 0.066 seconds.

A separate ninfer-serve OpenAI test returned exact SERVE_OK responses. The cold request reported 0 cached tokens; repeated non-streaming and streaming requests each reported 30 of 34 prompt tokens cached (88.2%).

Single-run rates can vary with prompt, output, driver state, sampling, context, and runtime revision. They establish that all four execution paths work; they are not a universal speedup guarantee.

Important limitations

• —Exact runtime matters: .ninfer is a registered, model-bound deployment format, not a generic container.
• —MTP specialization: the replacement MTP head is upstream-described as code-heavy; acceptance may differ on prose, multilingual, tool, or Vision workloads.
• —DFlash is text-only in NInfer: --spec dflash cannot be combined with --vision. Use MTP for multimodal speculative decoding.
• —Startup capabilities are fixed: requests cannot enable MTP, DFlash, or Vision weights that the server did not load at startup.
• —Only the documented 4k profile was tested here: larger contexts, higher concurrency, and CUDA Graphs need separate VRAM and stability validation.
• —Vision defaults are large: NInfer defaults to 8,192 Vision scratch tokens. Start with --vision-max-tokens 1024 on a 24 GB card and increase only when required.
• —No hosted inference: Hugging Face's hosted API cannot execute .ninfer files.

Conversion command

The artifact was produced with NInfer's dedicated converter:

.\.venv\Scripts\python.exe -m tools.convert.qwen3_6_35b_a3b.convert `
  --model '.\Ornith-1.5-35B-A3B-MTP' `
  --dflash-model '.\Qwen3.6-35B-A3B-DFlash' `
  --out '.\Ornith-1.5-35B-A3B-MTP.ninfer' `
  --device cuda

Conversion completed in 148.61 seconds after strict config, tensor-inventory, dtype, shape, and frontend-resource validation.

License and credits

The direct Shisa AI combined checkpoint is distributed under Apache-2.0 because its replacement MTP head derives from Qwen3.6. The retained Ornith target components remain MIT. The embedded Z-Lab DFlash component is Apache-2.0. See `NOTICE` and `LICENSES.md` for component-level attribution and pinned sources.

NInfer runtime code is not redistributed here; obtain it from its separately Apache-2.0-licensed upstream repository.

Converted, tested, and packaged by pyros-vault.