knoopx/ThinkingCap-Qwen3.6-27B-NInfer

ThinkingCap-Qwen3.6-27B for NInfer

A single-file NInfer (*.ninfer) artifact for ThinkingCap-Qwen3.6-27B, a token-efficient, thinking-oriented, multimodal (image+video→text) 27B finetune of `Qwen/Qwen3.6-27B`. The artifact is a self-contained binary that the NInfer engine serves directly; it is not a Transformers checkpoint, a Safetensors distribution, or a GGUF file.

This repository publishes exactly one model: the container-v3 ThinkingCap-Qwen3.6-27B-nvfp4-w8g32-q4g64-q5g64-q6g64-bf16.v3.ninfer artifact described below.

Model architecture

• —Qwen3_5ForCausalLM (model_type: qwen3_5), dense multimodal (image+video→text) finetune of `Qwen/Qwen3.6-27B`. Fully dense transformer: every decoder layer uses a dense SwiGLU MLP with no routing.
• —Text: 64 decoder layers; hybrid attention: 16 full-attention layers interleaved with 48 recurrent layers (short convolution, width 4), full-attention interval 4; hidden 5120, intermediate 17408, output matrix 248320 rows (tokenizer domain 248077); 24 query / 4 KV heads (head dim 256); 1 MTP draft layer; 262144 max position embeddings.
• —Vision: 27-layer ViT (hidden 1152, intermediate 4304, 16 heads), 16×16 spatial patch, 2-frame temporal patch, 2×2 spatial merge.
• —Positional: RoPE θ=10,000,000 with mRoPE section [11, 11, 10].
• —Upstream base: `bottlecapai/ThinkingCap-Qwen3.6-27B`, itself a finetune of `Qwen/Qwen3.6-27B`.

Artifact

The file contains Text, Vision, and MTP components plus frontend resources. Weight types are mixed across NVFP4 (W4A4, 247 tensors), the groupwise-integer formats Q4G64/Q5G64/Q6G64, W8G32 (vocabulary endpoints), and BF16/FP32.

Format histogram (sums to 1,301 tensors): BF16 591, FP32 343, NVFP4 247, Q4G64_F16S 55, Q5G64_F16S 54, W8G32_F16S 9, Q6G64_F16S 1, I32 1. Layouts: contiguous-le-v1 ×935, row-split-k128-v1 ×119, blockscale-k16-m128x4-v1 ×247.

Frontend resources

The embedded frontend uses the Qwen3.6 base native resources (tokenizer, tokenizer config, generation config, and the two vision preprocessor configs) together with the fixed Qwen chat template (froggeric/Qwen-Fixed-Chat-Templates, file chat_template.jinja). The fixed template adds developer-role handling, so serving requests that carry system/developer instruction turns no longer raise Unexpected message role.

Verify a downloaded file with:

printf '%s  %s\n' \
  '44e3186aee1e0551a79ebe15de2df50352ba204878b17ce3aabf0fba167bd760' \
  'ThinkingCap-Qwen3.6-27B-nvfp4-w8g32-q4g64-q5g64-q6g64-bf16.v3.ninfer' | sha256sum --check

Requirements

• —NInfer built from source (container-v3 reader/writer);
• —64-bit Linux;
• —NVIDIA GeForce RTX 5090 (sm_120a);
• —CUDA Toolkit 13.1 or newer.

NInfer does not provide an install target or packaged binary. See the repository README for source-build dependencies.

Download and run a CLI example

hf download knoopx/ThinkingCap-Qwen3.6-27B-NInfer \
  ThinkingCap-Qwen3.6-27B-nvfp4-w8g32-q4g64-q5g64-q6g64-bf16.v3.ninfer \
  --local-dir models

./build/apps/ninfer models/ThinkingCap-Qwen3.6-27B-nvfp4-w8g32-q4g64-q5g64-q6g64-bf16.v3.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 32768 \
  --max-new 8192 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

For images, videos, and structured chat history, see the CLI guide.

Start a local server

./build/apps/ninfer-serve \
  models/ThinkingCap-Qwen3.6-27B-nvfp4-w8g32-q4g64-q5g64-q6g64-bf16.v3.ninfer \
  --host 127.0.0.1 \
  --port 8080 \
  --max-context 240000 \
  --kv-capacity 240000 \
  --max-concurrency 2 \
  --kv-dtype fp8 \
  --device-state-slots 2 \
  --host-state-slots 8 \
  --host-kv-mib 8192 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

See the HTTP serving guide for the API surface and the resource scheduling reference for cache and admission semantics.

Supported use

The artifact supports:

• —text generation in thinking and non-thinking modes;
• —system, developer, user, and assistant message roles (fixed chat template);
• —image, multi-image, video, and mixed multimodal messages;
• —MTP speculative decoding with draft windows from one to five;
• —BF16, INT8, FP8, NVFP4, and K8V4 KV cache;
• —CUDA Graph decode and compatible-prefix reuse;
• —startup-bounded small-scale concurrent serving with true batched decode;
• —the NInfer CLI;
• —OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages serving.

Limits

• —NInfer executes on one RTX 5090 and one CUDA device, with a startup-fixed capacity of 1–8 active requests per Engine.
• —It does not provide large-scale or preemptive continuous batching, priority/QoS scheduling, multi-GPU execution, CPU/GPU offload, or distributed serving.
• —Context allocation is subject to GPU memory and the selected KV-cache type.
• —NInfer does not execute generated tool calls.

Provenance

The artifact identity, summarized object inventory, and provenance are published in `artifact-manifest.json`. The exact storage contract is maintained in the v3 container reference.

License

This NInfer artifact is distributed under the Apache License 2.0. The Qwen3.6-27B base and the ThinkingCap finetune are subject to their respective terms; users remain responsible for complying with the license and applicable laws.