soyaakinohara/Qwen-3.8-flash-next-heretic-3.78bpw-gguf

Qwen-3.8-Flash-Next Heretic 3.78bpw GGUF

A tight mixed-precision GGUF quantization (3.78 bpw) of trohrbaugh/Qwen3.8-Flash-Next-heretic-2, a refusal-attenuated (Heretic/abliterated) Qwen4Exp MoE derivative. Built from the BF16 source, calibrated with a Japanese-heavy importance matrix, and verified by hand-driven roleplay and generation tests.

The pipeline is: BF16 source → BF16 GGUF → imatrix calibration → architecture-aware mixed quantization. This release contains the final quantized text model plus the vision projector (mmproj).

Files

Qwen-3.8-flash-next-heretic-3.78bpw-00001-of-00002.gguf
Qwen-3.8-flash-next-heretic-3.78bpw-00002-of-00002.gguf
Qwen-3.8-flash-next-heretic-mmproj.gguf

Quantization recipe

Sensitive paths stay precise while the MoE bulk carries the compression:

MoE routed experts (outer 24 layers)  Q4_0
MoE routed experts (middle 24 layers) Q2_0
PLE ngram embeddings (102 GB)         Q4_0
GDN mixers (attn_gate/qkv, ssm_out)   Q4_K
Full-attention Q/O                    Q5_K
Shared experts                        Q5_0
Token embedding / output              Q6_K
GDN state (ssm_alpha/beta)            Q8_0
Routers / norms / scalars             F32 (kept, per llama.cpp policy)

Router tensors are kept in F32 and hyper-connection projections fall back to Q8_0 where K-quant block shapes do not fit. Both raise precision, not lower it.

llama.cpp usage

Upstream llama.cpp with qwen4exp support is sufficient (no fork needed). Single-turn CLI example (partial offload; full 83 GB exceeds 2×16 GB VRAM):

llama-cli -m Qwen-3.8-flash-next-heretic-3.78bpw-00001-of-00002.gguf -c 2048 -t 16 \
  -ngl 10 --tensor-split 1,1 --split-mode layer \
  -st -p "日本の首都は?"

Server example:

llama-server -m Qwen-3.8-flash-next-heretic-3.78bpw-00001-of-00002.gguf -c 8192 -t 16 \
  --host 127.0.0.1 --port 8899

llama.cpp resolves the -00002-of-00002 split automatically when the first file is given. For image input, add --mmproj Qwen-3.8-flash-next-heretic-mmproj.gguf.

Notes:

• —This is a thinking model. Responses start with [Start thinking] and end thinking with [End thinking] before the answer. Use max_tokens of 1024 or more; a 200-token budget gets consumed by thinking alone.
• —Tighter-than-usual quantization was chosen deliberately. If thinking loops or answers degrade on your workload, raise the middle-layer experts (see Provenance) or serve with reasoning disabled.

For a network-exposed server, add your own authentication, firewall, and access controls before exposing it beyond a trusted LAN.

Reported local performance

The model was prepared and tested on:

OS:       Ubuntu 24.04
GPUs:     NVIDIA GeForce RTX 5060 Ti 16GB x 2
RAM:      64GB + 23GB swap
Runtime:  llama.cpp CUDA build (upstream, commit ca3d5a3e1)

On this machine, short Japanese generation reached a reported 4.4 tokens/second at -ngl 10 partial offload. Actual speed depends on context length, prompt length, sampling settings, CUDA/llama.cpp version, and background workload. A tighter 3.26 bpw sibling was also built and rejected after hand testing (thinking loops, degraded answers); this 3.78 bpw build passed the same checks.

Provenance

Source model

• —trohrbaugh/Qwen3.8-Flash-Next-heretic-2
• —Source revision used locally: 207cd22c347a81e8d60e2f536c955da885641afb
• —Source precision: BF16, 34 safetensors files (~360 GB)
• —License: Qwen Community License 1.0 (full text)

Quantization steps

1. 1.Converted to BF16 GGUF (1224 tensors, ~354 GB) with convert_hf_to_gguf.py. Vision projector converted separately (--mmproj, 334 tensors).
2. 2.Calibrated an importance matrix: 80 chunks × 512 tokens, --process-output, on a Japanese-Wikipedia-heavy mix with English prose and code (~2.8 MB).
3. 3.Quantized with --imatrix and an exact-name --tensor-type-file (1224/1224 tensors mapped, 97 shape-driven fallbacks, all toward higher precision). Dry-run confirmed 3.78 bpw before the real run.
4. 4.Verified: GGUF header and per-tensor type histogram, deterministic short generation (Japanese factual answer, thinking trace present), and hand-driven roleplay evaluation by the publisher.

The local conversion and quantization used upstream llama.cpp (commit ca3d5a3e1, CUDA, 2-GPU).

AI assistance disclosure

The local model preparation workflow, conversion, quantization, validation, and this model card were performed with assistance from Hermes Agent. The model was then reviewed and published by the repository owner.

Responsible use

This is an uncensored / refusal-removed model. It may produce content that an aligned model would refuse, including unsafe, illegal, or harmful material. It has no reliable built-in safety layer. Use appropriate access controls, moderation, logging, and human review for any deployment, and comply with all applicable laws and policies.

The model is provided as-is. Users are responsible for prompts, outputs, and any downstream actions based on them.

日本語

概要

trohrbaugh/Qwen3.8-Flash-Next-heretic-2 の混合量子化 GGUF です。きつめの 3.78bpw で、 壊れやすい注意・GDN 状態系を高精度に残し、 パラメータの大半を占める MoE expert で圧縮しています。

無検閲化自体はソース側 (Heretic/abliteration) のもので、 量子化では日本語多めの重要度行列を使って感度を見ています。 思考ループの有無や回答品質は手動テストで確認しています。

使い方

qwen4exp 対応の llama.cpp があれば動きます (フォーク不要)。

llama-cli -m Qwen-3.8-flash-next-heretic-3.78bpw-00001-of-00002.gguf -c 2048 -t 16 \
  -ngl 10 --tensor-split 1,1 --split-mode layer \
  -st -p "日本の首都は?"

分割ファイルは 1つ目を指定すれば自動で読みます。 画像入力には --mmproj Qwen-3.8-flash-next-heretic-mmproj.gguf を足します。

thinking 系モデルのため、応答は [Start thinking] から始まります。 max_tokens は 1024 以上を推奨します。

注意

MTP ドラフトヘッドは含みません (llama.cpp の qwen4exp 変換の仕様)。 ライセンスは Qwen Community License 1.0 を継承します。 商用利用や大規模配備の条件はライセンス本文を確認してください。