mikeinnyc/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16

Qwen3.8-27B GPTQ INT4 symmetric G128 + unquantized MTP

Fresh GPTQ INT4 quantization of Qwen/Qwen3.8-27B, produced directly from the original BF16 checkpoint.

This is the G128 checkpoint selected from controlled G128 vs G32 testing on a single Intel Arc Pro B70 32 GB using vLLM XPU.

No quantization step is required when downloading this repository.

Full reproduction procedure, Dockerfiles, patches, benchmarks, calibration methodology, and deployment guide:

https://github.com/MikeCaldera/intel-arc-pro-b70-qwen38-vllm

Checkpoint

• —Base model: Qwen/Qwen3.8-27B
• —Quantization: GPTQ INT4
• —Group size: 128
• —Symmetric: true
• —desc_act: false
• —LM head quantization: disabled
• —MTP tensors: excluded from GPTQ quantization
• —MTP draft path used in testing: unquantized
• —Quantizer: GPTQModel 7.3.2
• —Tested serving stack: vLLM XPU
• —Tested GPU: Intel Arc Pro B70 32 GB

This is a quantized derivative, not a fine-tune. No additional training was performed.

Why G128

G128 and G32 were produced from the same BF16 source using the same frozen calibration token stream.

Under the tested gpu_memory_utilization=0.90 configuration with FP8 KV cache, G128 successfully served at a configured 161,000-token context on the Intel Arc Pro B70.

G32 required more model memory and did not provide enough remaining KV-cache capacity for the same 161K configuration. It also did not show enough source-fidelity improvement in the controlled tests to justify the additional memory use.

For this experiment, G128 is the preferred checkpoint.

Quantization calibration

Frozen calibration:

128 samples
1024 tokens per sample
131,072 calibration tokens total

Calibration file:

c4-fixed-128x1024.json

SHA256:

ddfc570e23458c048951501231c2ff75fa175440b120045bbeb1790bea5d2599

The calibration data was used only for GPTQ quantization calibration. It was not used to train or fine-tune the model.

Quantization procedure:

https://github.com/MikeCaldera/intel-arc-pro-b70-qwen38-vllm/blob/main/docs/QUANTIZATION.md

Tested Intel Arc Pro B70 serving profile

quantization = gptq
dtype = float16
max_model_len = 161000
gpu_memory_utilization = 0.90
kv_cache_dtype = fp8
max_num_seqs = 1
max_num_batched_tokens = 8192
prefix caching = disabled
speculative decoding = MTP
num_speculative_tokens = 4

Environment used during testing:

B70_MTP_BF16_DRAFT=1
VLLM_WORKER_MULTIPROC_METHOD=spawn
VLLM_XPU_ENABLE_XPU_GRAPH=1
VLLM_TARGET_DEVICE=xpu
ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE
ZE_AFFINITY_MASK=0
PYTORCH_ALLOC_CONF=expandable_segments:True

The tested B70 MTP4 configuration also uses the vLLM XPU patches documented in the GitHub repository.

Downloading this checkpoint removes the need to perform GPTQ quantization yourself, but the tested B70 MTP4 serving path still requires the documented vLLM XPU environment and patches.

Deployment guide:

https://github.com/MikeCaldera/intel-arc-pro-b70-qwen38-vllm/blob/main/docs/B70-QUALITY-DEPLOYMENT.md

Download

This is not a single GGUF file.

The checkpoint is stored as five .safetensors model shards plus tokenizer and configuration files. Download the entire repository, not an individual shard.

Using the Hugging Face CLI:

hf download mikeinnyc/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 \
  --local-dir ~/models/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16

The complete download is approximately 19.6 GB on Hugging Face.

After download, the directory should contain files including:

model-00001-of-00005.safetensors
model-00002-of-00005.safetensors
model-00003-of-00005.safetensors
model-00004-of-00005.safetensors
model-00005-of-00005.safetensors
model.safetensors.index.json
config.json
quantize_config.json
tokenizer.json
tokenizer_config.json
chat_template.jinja

Run with vLLM + Open WebUI

This checkpoint is intended to be served by vLLM and then accessed through an OpenAI-compatible client such as Open WebUI.

It is not loaded directly into Open WebUI.

The basic flow is:

Hugging Face checkpoint
        |
        v
      vLLM
        |
        v
OpenAI-compatible API
        |
        v
    Open WebUI

1. Download the model

hf download mikeinnyc/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 \
  --local-dir ~/models/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16

2. Serve it with vLLM XPU

For Intel Arc Pro B70, use the tested Docker/vLLM XPU configuration documented here:

https://github.com/MikeCaldera/intel-arc-pro-b70-qwen38-vllm/blob/main/docs/B70-QUALITY-DEPLOYMENT.md

The reference configuration exposes vLLM on host port:

11444

with the OpenAI-compatible API available at:

http://127.0.0.1:11444/v1

The reference served-model name is:

qwen38

Verify that vLLM is running:

curl http://127.0.0.1:11444/v1/models

You should receive a JSON response containing the available model.

3. Connect Open WebUI

If you already use Open WebUI with Ollama or another backend, you can add vLLM as an additional OpenAI-compatible connection.

In Open WebUI, add an OpenAI-compatible API connection.

Open WebUI running directly on the host

Use:

API URL:
http://127.0.0.1:11444/v1

Open WebUI running in Docker and vLLM exposed on the host

Inside an Open WebUI container, localhost refers to the Open WebUI container itself.

Use a host-reachable address such as:

http://host.docker.internal:11444/v1

On Linux Docker hosts, host.docker.internal may need to be explicitly mapped to the host gateway.

For example, the Open WebUI container can be started with:

--add-host=host.docker.internal:host-gateway

Open WebUI and vLLM on the same Docker network

If both containers share a Docker network, Open WebUI can connect directly to the vLLM container using its container name and internal port.

For the reference container:

http://qwen38-quality-mtp4:8000/v1

This avoids routing the connection back through the host port.

4. Select the model

After the vLLM connection is saved, the model served as:

qwen38

should appear in Open WebUI's model selector.

Select it and use Open WebUI normally.

Important difference from GGUF / llama.cpp

Users coming from llama.cpp or Ollama may be accustomed to downloading a single .gguf file.

This model is different.

It uses a Hugging Face GPTQ checkpoint consisting of multiple .safetensors shards:

5 model shards
+ model index
+ tokenizer
+ GPTQ configuration
+ model configuration

Do not download only one .safetensors shard.

Download the entire repository and point vLLM at the resulting model directory.

B70 users

For the exact Intel Arc Pro B70 Docker image, MTP4 patches, XPU environment variables, FP8 KV configuration, and 161K-context launch command, use:

https://github.com/MikeCaldera/intel-arc-pro-b70-qwen38-vllm/blob/main/docs/B70-QUALITY-DEPLOYMENT.md

That is the tested configuration used for the published B70 results.

Quality validation

A 30-prompt source-fidelity suite tested conflicting documents, missing information, timeline ambiguity, document hierarchy, source provenance, and unsupported explanations.

Final deterministic profile:

temperature = 0.0
top_p = 1.0
top_k = -1
max_tokens = 4096

Two back-to-back deterministic G128/MTP4 runs completed:

30/30 requests completed
0 errors
0 truncations

Aggregate generation throughput on this thinking/source-fidelity workload was approximately:

57.75 tok/s

This is not directly comparable to the separately published 84.65 tok/s short-context decode benchmark because the workloads are different.

Quality methodology and published result files:

https://github.com/MikeCaldera/intel-arc-pro-b70-qwen38-vllm/tree/main/benchmarks/quality

Known limitation

One conflicting-document test repeatedly produced a disputed interpretation involving quantities of 240 and 238.

The model treated the receiving log's 238-unit figure as definitively received rather than leaving the conflicting quantities unresolved.

This behavior was observed across G128, G32, multiple prompt variants, sampled inference, and deterministic inference. There is therefore no evidence that it is simply caused by GPTQ group size.

The benchmark itself also contains semantic ambiguity: an invoice quantity and a receiving-log quantity can represent different facts. Future testing should explicitly distinguish fields such as units shipped and units received.

This limitation is documented intentionally rather than presenting the checkpoint as perfect source fidelity.

Recommended high-reliability RAG flow

For high-stakes document QA:

documents
  -> extract facts
  -> normalize fields
  -> detect conflicts
  -> classify evidence
  -> construct grounded context
  -> LLM generation

Do not rely exclusively on prompt engineering to detect every evidence conflict.

Performance reference

Separate short-context MTP4 benchmark:

Median decode: 84.65 tok/s
Mean decode:   84.49 tok/s
Prompt:        512 tokens
Generation:    128 tokens

Full methodology and evidence:

https://github.com/MikeCaldera/intel-arc-pro-b70-qwen38-vllm

Credits

Qwen

Original model:

Qwen/Qwen3.8-27B

https://huggingface.co/Qwen/Qwen3.8-27B

GPTQModel

GPTQModel 7.3.2 was used for the fresh GPTQ INT4 quantization.

vLLM / Intel XPU

vLLM XPU was used for serving, long-context validation, and MTP speculative decoding testing.

SergiioB

Special thanks to SergiioB and the Intel Arc Pro B70 inference cookbook for the B70 vLLM/XPU/MTP groundwork:

https://github.com/SergiioB/intel-arc-pro-b70-inference-cookbook

Reproduction and quantization

Fresh quantization, controlled G128/G32 testing, calibration freezing, B70 validation, quality benchmarking, and documentation:

Mike Caldera

https://github.com/MikeCaldera/intel-arc-pro-b70-qwen38-vllm

License

This checkpoint is a quantized derivative of Qwen3.8-27B.

See the upstream model repository and license for the original model:

https://huggingface.co/Qwen/Qwen3.8-27B

Disclaimer

Performance and maximum context capacity depend on the complete hardware and software environment.

The 161K configuration is a measured result from the tested Intel Arc Pro B70 system and is not a guarantee for every installation.