Modelpublic

lyf/Qwen3.8-27B-Huihui-Abliterated-NInfer-NVFP4

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

15likes7.2kdownloads

Model Card

Huihui Qwen3.8-27B Abliterated — NInfer Mixed FP8/NVFP4

A complete NInfer artifact derived from huihui-ai/Huihui-Qwen3.8-27B-abliterated revision d42ca8978c5a66e92c3446d46e8adfe03ef692ff.

The artifact combines NInfer's fixed FP8 + NVFP4 Qwen3.8 allocation with same-source Huihui BF16 norms, embeddings, vision tower, and MTP tensors. No official Qwen or unrelated behavioral checkpoint supplies model weights.

Download and run

bash

hf download lyf/Qwen3.8-27B-Huihui-Abliterated-NInfer-NVFP4 qwen3_8_27b_nvfp4.ninfer   --local-dir ./huihui-ninfer

git clone https://github.com/Neroued/ninfer.git
cd ninfer
git checkout a99407c63fc5bbd25d9fb597cbb8ab352bdb01ef
cmake -S . -B build -G Ninja   -DCMAKE_BUILD_TYPE=Release   -DCMAKE_CUDA_ARCHITECTURES=120a
cmake --build build -j"$(nproc)"

./build/apps/ninfer ../huihui-ninfer/qwen3_8_27b_nvfp4.ninfer   --prompt "Explain FP8 and NVFP4 briefly."   --max-new 128 --max-context 8192 --kv-dtype int8 --no-thinking

OpenAI-compatible server

bash

./build/apps/ninfer-serve ../huihui-ninfer/qwen3_8_27b_nvfp4.ninfer   --host 127.0.0.1 --port 8000   --model-id qwen38-huihui-ninfer-nvfp4   --max-context 204800 --kv-capacity 204800   --max-concurrency 1 --prefill-chunk 4096   --kv-dtype int8 --spec mtp --draft-tokens 3   --default-max-tokens 16384 --preserve-thinking

This 204.8K server profile was startup- and API-smoke-tested on one RTX 5090. Long-context quality and maximum sustained prompt length remain workload-dependent.

Mixed allocation

Matrices	Format
Full attention and Gated DeltaNet projections	FP8 E4M3, per-output-row BF16 scales
`lm_head` and layers 56–63 MLP	FP8 E4M3
Layers 0–55 MLP	NVFP4, group size 16
Norms, GDN state, embeddings, vision, MTP	Same-source Huihui tensors

NInfer source preflight passed with:

text

FP8 source matrices: 233
NVFP4 source matrices: 168
source fields: 1587
F8_E4M3: 401
BF16: 682
U8: 168
F32: 336
NINF_PREFLIGHT_OK

The FP8 matrices are deterministic row-scaled E4M3 exports from Huihui BF16. The NVFP4 matrices come from the separately calibrated Huihui ModelOpt NVFP4 checkpoint (CNN/DailyMail 3.0.0, 20 × 8192 tokens).

Artifact inventory

text

NInfer commit: a99407c63fc5bbd25d9fb597cbb8ab352bdb01ef
model_id: qwen3.8-27b
weights_id: nvfp4
objects: 1124 (1118 tensors, 6 resources)
artifact bytes: 21,492,695,040
SHA256: f21f308d3b23ccd627071cd015e413db08deee4356643900518e2b251750fdc2

Key formats:

text

BF16: 534
FP32: 208
FP8 row-scaled: 146
NVFP4: 112

Runtime validation

Hardware: RTX 5090 / SM120, driver 610.43.02, CUDA 13.1, 450 W cap.

Text smoke test:

text

prompt tokens: 27
generated tokens: 114
prefill: 792.69 tok/s
decode: 74.83 tok/s
overall: 73.83 tok/s
GPU weights: 18.98 GiB

Real-image test passed. For the test image, the model correctly identified a red square in the upper-left and a blue circle in the lower-right. Vision decode measured 74.82 tok/s with 19.25 GiB of GPU weights loaded.

A benign ablation/refusal-vector research prompt also received a direct technical answer. This is a smoke test of behavioral continuity, not a comprehensive behavioral benchmark.

Included evidence

—qwen3_8_27b_nvfp4.ninfer.conversion.json
—ninfer-nvfp4-preflight.log
—artifact-inspect.log
—text, vision, and behavior outputs/logs
—BUILD_MANIFEST.json, VALIDATION_REPORT.json, SHA256SUMS

Intended use

This is an abliterated behavioral derivative intended for model research and local inference. Users are responsible for downstream use and applicable policies.