IndrasMirror/Swift-Qwen3.8-27B-Uncensored-W4A16

Swift-Qwen3.8-27B-Uncensored-W4A16 (AutoRound)

The uncensored "Swift" Qwen3.8-27B checkpoint, quantized from FP8 to a 4-bit W4A16 compressed-tensors format with AutoRound, and tuned to serve on vLLM with DFlash2 speculative decoding on a 24 GB card (RTX 4090 / 3090 class) at a 200,000-token context with int4 KV cache.

This is the model that powers the qwen3.8-swift-quetza-vllm local launch profile. It is a direct sibling of the 4-bit base profile — same architecture, same KV scheme, same drafter — so the only variable is the (uncensored) weights.

Ungated — freely downloadable. The upstream sources (`d0xin/Swift-Qwen3.8-27B-Uncensored-FP8`, `ukisai/Swift-Qwen3.8-27b`) are gated by their authors; this derivative is not.

Model summary

The base is the uncensored / abliterated derivative of `ukisai/Swift-Qwen3.8-27b` (directional residual-stream ablation), itself a Qwen3.8-27B. Expect the refusal-reduced behavior of that family — see Safety.

Build recipe

Reproduced with AutoRound (llm-compressor) from the FP8 source:

auto-round \
  --model  <Swift-Uncensored-FP8> \
  --scheme W4A16 \
  --dataset NeelNanda/pile-10k \
  --nsamples 128 --seqlen 2048 --batch_size 4 \
  --iters 200 --seed 42 \
  --format llm_compressor

Post-step: embed_tokens and lm_head are packed from BF16 to 4-bit pack-quantized int4 (group 128, symmetric, per-group scale) to match the working base's layout. The packing script dequantizes each tensor it packs and aborts the run if the reconstruction cosine falls below 0.99, so the committed tensors necessarily passed that gate (the per-tensor figures themselves were not retained).

AutoRound itself does not target nn.Embedding (auto_round/utils/model.py — supported_types = (nn.Linear, Conv1D)), so the embedding/head packing is a separate surgical step. If you re-quantize, run it afterwards, or these two tensors stay BF16 (2.54 GB each).

Memory & why this fits a 24 GB card

The decisive detail: `embed_tokens` and `lm_head` are 4-bit packed (1.27 GB each), not BF16. That is exactly the layout of the working base profile, and it is what lets the whole thing load with the DFlash2 drafter on 24 GB. A checkpoint where those two are left in BF16 is ~2.5 GB heavier and OOMs at load.

This Swift release sits above the base because the uncensored source carries a heavier language model (17.68 vs 15.18 GB). The 200k int4 KV cache and the DFlash2 drafter are what consume the remaining VRAM at runtime; the serving wrapper (--ctx, --no-draft) trades those down if you are short.

Running it

The companion launcher (a docker/vLLM wrapper, not in this repo) is `qwen3.8-swift-quetza-vllm`:

qwen3.8-swift-quetza-vllm                       # start vLLM + QuetzaCodetl
qwen3.8-swift-quetza-vllm --no-draft            # if the load OOMs (frees the drafter)
qwen3.8-swift-quetza-vllm --no-reasoning        # thinking OFF (server-side, restarts)
qwen3.8-swift-quetza-vllm --effort=medium       # low | medium | xhigh (default xhigh)
qwen3.8-swift-quetza-vllm --ctx=131072          # reduce context to fit more headroom
qwen3.8-swift-quetza-vllm --status              # backend state
qwen3.8-swift-quetza-vllm --stop                # free the GPU

It uses the image ghcr.io/syv-ai/qwen38-27b-rtx3090:latest, serves on localhost:18022 as model name qwen3.8-27b, int4 KV cache, 200K context.

If you would rather run it yourself, a minimal vLLM bring-up is:

vllm serve /path/to/Swift-Uncensored-W4A16 \
  --served-model-name qwen3.8-27b \
  --tensor-parallel-size 1 \
  --max-model-len 200000 \
  --kv-cache-dtype int4_per_token_head \
  --trust-remote-code

Recommended generation settings

Carried over from the Qwen3.8 / Swift release card.

Thinking mode (reasoning, coding, agentic):

medium and low are valid; high is rejected (HTTP 400) by the tested stack. For agentic work, lower effort does not always cut total latency — insufficient reasoning can add retries.

Instruct / non-thinking mode:

Files

• —model-0000{1..7}-of-00007.safetensors — quantized weights (7 shards)
• —model_extra_tensors.safetensors — MTP / extra tensors (kept out of the index; the launcher uses them as needed)
• —model.safetensors.index.json — weight map (2003 tensors)
• —config.json, quantization_config.json, generation_config.json
• —tokenizer.json, tokenizer_config.json, chat_template.jinja
• —preprocessor_config.json, processor_config.json — vision processor

Safety

This is an uncensored / refusal-reduced derivative. It may generate content the upstream model would refuse, restrict, or handle more cautiously. Outputs can be inaccurate, offensive, or otherwise unsuitable for a given use. It is not intended to provide professional legal, medical, financial, safety-critical, or other regulated advice. Use at your own discretion and risk.

License

Swift weights are distributed through gated access under the Swift Open License v1.0. Personal, research, educational, evaluation, and commercial use are free for individuals and organizations with annual recurring revenue (including affiliates) of up to US$1,000,000. Above that, commercial use requires a separate Swift Enterprise License from UkisAI.

Attribution

Original model: UkisAI, `ukisai/Swift-Qwen3.8-27b`. Uncensored / FP8 source: `d0xin/Swift-Qwen3.8-27B-Uncensored-FP8`. W4A16 AutoRound quantization, embedding/head packing, and vLLM serving profile packaged by IndrasMirror.