Schestex/ThinkingCap-Qwen3.8-27B-NInfer
ThinkingCap-Qwen3.8-27B — NInfer D48 NVFP4
Experimental NInfer conversion of ThinkingCap-Qwen3.8-27B, optimized for NVIDIA Blackwell GPUs with a quality-first mixed-precision strategy.
The current recommended artifact is the D48 anchor.
It starts from NInfer's standard qwen3_8_27b recipe and adds model-specific NVFP4 quantization to selected MLP Down projections. The D48 layers were selected from a complete 64-layer single-change sensitivity sweep and then validated cumulatively with full perplexity and runtime benchmarks.
Artifact
Place the model file in the repository root with this name:
thinkingcap-qwen3.8-27b-d48.ninferThe corresponding standard NInfer baseline artifact was 17,106,573,056 bytes. D48 reduces the artifact by 401,043,708 bytes (~382.5 MiB / 2.34%).
Artifact size note: This build is not directly size-comparable to the full upstream Qwen3.8 NInfer/NVFP4 artifacts. It was converted with --components text and currently does not bundle Vision, MTP or DFlash2 companion weights. D48 also retains NInfer's compact groupwise allocation for most tensors and selectively replaces only 48 MLP Down projections with NVFP4. As a result, the optimized artifact is smaller than its own 17.11 GB text-only NInfer baseline. The smaller size does not imply that this build uses a more aggressive full-model NVFP4 profile. It is a selective mixed-precision configuration optimized around the measured ThinkingCap sensitivity map.
What D48 means
D48 does not mean “layers 0–47”. Every one of the 64 MLP Down projections was tested individually. The 48 most tolerant layers were selected for the cumulative D48 build.
NVFP4 MLP Down layers
0, 1, 4, 5, 6, 7, 9,
13, 14, 15, 16, 17, 19, 20, 22, 23, 24,
28, 29, 30, 31,
33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
53, 54, 55, 56, 58, 59, 60, 61, 62Protected MLP Down layers
2, 3, 8, 10, 11, 12, 18, 21,
25, 26, 27, 32,
51, 52, 57, 63These remain at the precision selected by the standard NInfer recipe.
Quantization strategy
New artifacts are generated directly from the original BF16/FP16 ThinkingCap checkpoint. Existing .ninfer files are not requantized.
ThinkingCap BF16/FP16 Safetensors
│
▼
NInfer qwen3_8_27b recipe
│
▼
standard mixed-precision NInfer baseline
│
▼
selected MLP Down projections → NVFP4
│
▼
ThinkingCap D48 NInferD48 is still a mixed-precision model. Only the selected MLP Down projections receive the additional NVFP4 override; all other tensors continue to use the formats chosen by the standard NInfer recipe.
Full perplexity validation
Development corpus configuration:
context: 4096
stride: 2048
KV cache: BF16
scored tokens: 782,535
streams: 12The corpus contains English long-form text, English reference text, and code. It is a development/sensitivity corpus, not an untouched final holdout.
Overall D48 delta versus the baseline is approximately −0.070% PPL on this development corpus.
The negative aggregate delta must not be interpreted as evidence that quantization improves the underlying model. The individual domains move in different directions; the result is best read as no measurable aggregate quality loss on this development corpus at the tested resolution.
Runtime benchmark
Measured on an NVIDIA RTX PRO 4000 Blackwell 24 GB with FP8 KV cache, prefill chunk 1024, 2 warmups and 5 measured repetitions.
These values are intended for comparisons within the same NInfer/hardware/software setup. They should not be treated as directly comparable to unrelated runtimes or benchmark methodologies.
Why D48 is the current anchor
More aggressive cumulative candidates were also tested:
D50 was only about 16 MiB smaller than D48 and measured roughly +0.31% mean prefill / −0.12% mean decode versus D48 in the same benchmark setup. D53 produced a clearer prefill gain, but also a clearer perplexity regression.
For the current quality-first target, D48 is therefore the working Down anchor.
Current optimization status
1. Standard NInfer conversion
2. Full 64-layer MLP Down sensitivity sweep
3. D48 quality-first Down anchor ← this release
4. ThinkingCap-specific IMatrix
5. Gate / Up sensitivity analysis
6. Combined mixed-precision optimization
7. Final task-level quality validation
8. MTP and/or DFlash2 integrationNot yet included
This D48 artifact does not yet include:
- ThinkingCap-specific IMatrix optimization
- Agentic-workload importance calibration
- dedicated Gate optimization
- dedicated Up optimization
- final Gate/Up + Down joint optimization
- MTP speculative decoding model
- DFlash2 speculative decoding model
- final task-level evaluation suite
Planned IMatrix work
A ThinkingCap-specific importance matrix is planned from the original BF16/FP16 weights using an agentic/code-oriented calibration corpus.
The IMatrix will be used to prioritize Gate/Up candidates, identify important tensors/channels, reduce the search space, and cross-check the existing empirical sensitivity results. It will not replace direct single-change and cumulative-group quality measurements.
Reproducibility
Converter checkout base commit:
9e163eee4b8acec21ab0ac765107b6a3f287b217The local build also contains experimental changes required for the NVFP4 path, so exact compatibility can depend on the corresponding NInfer runtime implementation.
See PROVENANCE.md, quantization/d48-layers.txt, and benchmarks/ for compact experiment metadata.
Intended use
This release is primarily intended for:
- ThinkingCap-Qwen3.8-27B experiments with NInfer
- NVIDIA Blackwell inference
- mixed-precision / NVFP4 research
- quality/performance trade-off analysis
- further model-specific optimization work
The optimization priority is:
1. Preserve model quality
2. Improve real inference throughput
3. Reduce persistent model/VRAM footprint
4. Keep experiments reproducibleLicense and upstream terms
This repository does not redefine the licenses of the upstream ThinkingCap model, its base model, NInfer, or NVIDIA components. Users should review and comply with the applicable upstream licenses and terms before redistribution or use.