jasonboukheir/Qwen3.8-27B-AEON-Ultimate-Uncensored-BF16-W4A16-AutoRound

Qwen3.8-27B AEON Ultimate Uncensored — W4A16 AutoRound

This repository contains a W4A16 quantization of `AEON-7/Qwen3.8-27B-AEON-Ultimate-Uncensored-BF16`, exported as compressed-tensors for vLLM. An earlier revision enabled experimental dataset-calibrated static FP8 KV-cache scales by default. Testing found that cache unsafe, so this revision deliberately omits kv_cache_scheme and lets auto use the model's native BF16 cache.

The quantized language model is combined with the exact BF16 vision tower and BF16 MTP tensors from the pinned source revision. These tensors were grafted without conversion after quantization because Transformers loads this model's text backbone separately from its conditional-generation wrapper.

The quantization was produced for Intel XPU serving. Other hardware and serving engines may work if they support this compressed-tensors scheme, but have not been validated for this release.

Reproducibility

The source revision was resolved and pinned before calibration. Weight quantization and the retained diagnostic FP8 KV-scale calibration used the same aligned calibration dataset.

Full-precision exclusions

The following model-specific selectors were excluded from AutoRound weight quantization:

• —lm_head
• —token embeddings (re:.*embed_tokens$; embeddings are not Linear targets)
• —DeltaNet/linear-attention A and B input projections (re:.*linear_attn\.in_proj_[ab]$)

The exported config.json contains the fully resolved compressed-tensors ignore list. These exclusions apply to weight quantization. The historical model-kv-scales.safetensors shard and calibration recipe remain for diagnosis, but config.json intentionally has no kv_cache_scheme, so serving software does not select FP8 automatically.

Serving with vLLM

Use the backend's native cache dtype. This configuration has been tested on Intel XPU:

vllm serve jasonboukheir/Qwen3.8-27B-AEON-Ultimate-Uncensored-BF16-W4A16-AutoRound \
  --quantization compressed-tensors \
  --kv-cache-dtype auto \
  --max-model-len 65536

For this BF16 model, the absence of kv_cache_scheme makes auto select the backend's native BF16 KV cache. Do not serve this revision with `--kv-cache-dtype fp8`. A long-generation test using the retained calibrated static scales degenerated into severe token and phrase repetition. Repeating the test with the same quantized weights and a BF16 KV cache produced coherent output. This isolates the observed failure to the FP8 cache path or its scales rather than demonstrating a W4A16 weight failure.

The inactive FP8 scale shard remains in the repository for reproducibility and diagnosis, not as a serving recommendation. It needs recalibration plus layerwise and long-context parity evaluation before a future revision restores FP8 metadata.

The source architecture combines linear-attention and periodic full-attention layers. Confirm that your vLLM version supports this Qwen architecture and the chosen XPU kernels.

The restored Qwen3_5ForConditionalGeneration wrapper supports image inputs as well as text. Image preprocessing and OpenAI-compatible request syntax depend on the serving version; consult the matching vLLM multimodal documentation.

Validation status

Completed checks:

• —the quantization run completed successfully;
• —the output was serialized as compressed-tensors;
• —W4A16 metadata is present in config.json;
• —config.json omits FP8 KV metadata, so auto selects native BF16;
• —the inactive model-kv-scales.safetensors diagnostic shard is retained;
• —the published artifact manifest records SHA-256 hashes for every file;
• —the BF16 MTP and vision tensors were restored from the pinned source and the conditional-generation checkpoint loaded successfully;
• —vLLM selected its Intel XPU W4A16 compressed-tensors kernel;
• —MTP drafting, compiled XPU graphs, multimodal warmup, and a 65,536-token serving window initialized successfully;
• —generation with a BF16 KV cache was coherent;
• —generation with the included calibrated FP8 KV scales exhibited severe repetitive degeneration and is not recommended;
• —the complete grafted checkpoint is approximately 18.12 GiB.

Not yet reported for this release:

• —perplexity or downstream quality evaluation;
• —generation parity against the BF16 source;
• —systematic long-context and layerwise KV-cache sensitivity evaluation;
• —throughput, latency, or memory benchmarks.

No quality or speed improvement over the source model is claimed until those tests are published.

Intended use and limitations

This is a quantized derivative of the source model. Its behavior, intended use, training history, safety characteristics, and content limitations originate with the source model. Quantization may introduce additional quality loss or behavior changes.

The source is described as uncensored. It may generate inaccurate, harmful, offensive, or otherwise objectionable content. Do not treat its output as professional, legal, medical, financial, or safety-critical advice. Operators are responsible for appropriate evaluation, access controls, and safeguards.

License

No independent license is asserted by this quantization repository. Review and comply with the license and terms published by the `AEON-7/Qwen3.8-27B-AEON-Ultimate-Uncensored-BF16` source repository and any upstream base models before using or redistributing these weights.

Quantization tooling

Created with the reproducible workflow in `jasonbk/vllm-xpu-nix`, using llm-compressor's AutoRound modifier and a pinned Intel XPU toolchain.