machine-made-Fibre/qwen3-8b-heretic-OpenVINO-INT4-NPU

qwen3-8b-heretic OpenVINO INT4 for Intel NPU

This repository contains a ready-to-run OpenVINO IR export of `DreamFast/qwen3-8b-heretic`, prepared for local inference on Intel NPU through OpenVINO Model Server.

It is intended for users who want to skip the local OpenVINO conversion and weight compression steps.

Source model

• —Source model: DreamFast/qwen3-8b-heretic
• —Original base model: Qwen/Qwen3-8B
• —Architecture: Qwen3ForCausalLM
• —Task: text generation
• —License: Apache-2.0, inherited from the source model metadata

This is not a fine-tune. It is an OpenVINO INT4 runtime export of the source model above.

OpenVINO export

The exported directory includes OpenVINO model IR files, OpenVINO tokenizer and detokenizer files, tokenizer files, chat template, and generation config.

Compression metadata from this export:

{
  "mode": "INT4_SYM",
  "group_size": -1,
  "ratio": 1.0,
  "all_layers": true
}

The model config reports max_position_embeddings: 40960.

Tested Intel NPU runtime

Prepared on Windows for OpenVINO / OVMS text generation with:

• —Target device: NPU
• —OVMS task: text_generation
• —Recommended max concurrent sequences: 1
• —Recommended cache interval multiplier: 64

Example OVMS command:

ovms.exe `
  --model_path Q:/llm/models/OpenVINO/DreamFast--qwen3-8b-heretic-text-fp16-true-int4-sym-cw-ov `
  --model_name DreamFast--qwen3-8b-heretic-true-int4-npu `
  --rest_port 8000 `
  --rest_bind_address 0.0.0.0 `
  --task text_generation `
  --target_device NPU `
  --max_prompt_len 8192 `
  --max_num_seqs 1 `
  --cache_interval_multiplier 64 `
  --reasoning_parser qwen3 `
  --tool_parser hermes3

Local benchmark

This 8B export was uploaded from a local converted OpenVINO INT4 directory, but it was not separately benchmarked in the upload session. Do not infer token speed from file size.

Known local artifact size:

• —openvino_model.bin: about 3.82 GiB

If loading stalls or system memory pressure is too high, reduce --max_prompt_len.