WatchDG/Qwen3.8-27B-ABLITERATED-EXL3-3.75bpw

Qwen3.8-27B-ABLITERATED-EXL3-3.75bpw

by WatchDG

EXL3 quantization of Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16 (commit 9d85770e5eb602322b4bceef55beda357e0bd0ca), an abliterated Qwen3.8-27B multimodal model.

Third-party derivative, unevaluated by upstream. This is a community EXL3 build, not an official Qwen or Blackfrost release. Blackfrost's own model card states: "Any additional direction editing, fine-tuning, merging, pruning, quantization, or other weight change creates an artifact Blackfrost has not evaluated unless a new report explicitly states otherwise." That applies directly here — quality, refusal behavior, reasoning, tool use, vision/video behavior, and long-context stability of this specific 3.75 bpw quant have not been assessed by Blackfrost and may differ from the BF16 reference. This is a weight-level research checkpoint with a deliberately reduced refusal surface; it is not a safety-stock model and must not be represented, deployed, or evaluated as one.

• —Architecture: Qwen3_5ForConditionalGeneration (hybrid linear + full attention, 64 layers, MTP)
• —Parameters: ~27.78 B (text + vision)
• —Native context: 262 144 tokens
• —Vocab: 248 320 (Qwen3.8)
• —Quantization: EXL3 v1.4.2
• —Disk size: 16.10 GB (2 shards × ~8 GB)

Quantization

• —Codebook: mul1 (default)
• —Out scales: always
• —Calibration: exllamav3 v1.4.2 standard data (c4/code/multilingual/technical/wiki/tiny), 250 rows × 2048 cols
• —Reported per-tensor SQNR: median 35.0 dB, min 29.1 dB, max 46.0 dB (479 tensors; norm/MTP excluded as their quantiser printout shows sqnr=0)

Effective average over the whole file (including head/MTP/vision): ~4.64 bpw (16.10 GB × 8 / 27.78 B params).

Files

Modifications vs upstream

The repo files match the Blackfrost source byte-for-byte except for three intentional additions:

1. 1.`config.json` — appended a quantization_config block (ExL3 specifics). All upstream fields preserved unchanged.
2. 2.`quantization_config.json` — exllamav3-generated per-tensor storage table; not present in the BF16 source (which has no quantization).
3. 3.`chat_template.jinja` — added or message.role == "developer" in three places so that the template also accepts role=developer as an alias for role=system. The upstream template only recognises system/user/assistant/tool; the developer role is used by some OpenAI-style APIs for the higher-priority instruction.

All other sidecar files (generation_config.json, tokenizer.json, tokenizer_config.json, preprocessor_config.json, video_preprocessor_config.json) are byte-identical to the Blackfrost source.

Quick start (TabbyAPI + exllamav3)

Full tabby-config-3.75.yml used to serve this model on a 24 GB GPU (RTX 4090 verified):

network:
  host: 0.0.0.0
  port: 5000
  disable_auth: true       # only safe behind localhost / a trusted network

model:
  model_dir: /app/models
  model_name: Qwen3.8-27B-ABLITERATED-EXL3-3.75bpw

  # --- KV cache (24 GB budget) ---
  cache_size: 131072       # max tokens of KV cache
  cache_mode: Q8           # 8-bit cache; halves VRAM vs FP16, < 10% speed cost
  max_seq_len: 131072      # cap request length to the cache

  # --- EXL3-specific (optional) ---
  chunk_size: 2048         # prompt ingestion chunk size
  output_chunking: true    # allocate KV cache in chunks as needed (saves VRAM)
  inline_model_loading: true   # allow model swap by name in request body

  # --- multi-GPU / single-GPU split ---
  gpu_split_auto: true
  autosplit_reserve: [96]  # MiB reserved for activations on the single GPU

  # --- chat template / parsing ---
  prompt_template:         # use the model's embedded template
  template_vars_default:
    enable_thinking: true  # reasoning on by default
  reasoning: true          # required for tool call parsing; splits reasoning_content / content
  vision: true             # model is multimodal (Qwen3.5 vision tower)

  # --- tool / function calling parser ---
  tool_format: qwen3_coder
  tool_calls_in_reasoning: true

  # --- model-name aliasing ---
  use_dummy_models: true
  dummy_model_names: ["qwen3.8-27b"]   # short alias exposed via /v1/models

draft_model:
  draft_mode: disabled     # no speculative decoding

sampling:
  override_preset: safe_defaults   # provides sampling fallbacks for OAI clients that don't send params

logging:
  log_prompt: false
  log_generation_params: false
  log_requests: false
  log_chat_completion_requests: false

memory:
  cuda_malloc_async: true  # torch allocator; recommended for EXL3

developer:
  unsafe_launch: false

Note: the ghcr.io/theroyallab/tabbyapi image lacks python3.12-dev, which Triton needs for JIT compilation of cuda_utils.c. The entrypoint is overridden to install the dev headers and start the server in one shot:

docker run --rm --gpus all -p 10404:5000 \
  -v './Qwen3.8-27B-ABLITERATED-EXL3-3.75bpw:/app/models/Qwen3.8-27B-ABLITERATED-EXL3-3.75bpw:ro' \
  -v './tabby-config-3.75.yml:/app/config.yml:ro' \
  --entrypoint bash ghcr.io/theroyallab/tabbyapi:latest \
  -c "apt-get update -qq && apt-get install -y --no-install-recommends python3.12-dev gcc && exec python3 main.py --host 0.0.0.0"

VRAM observed during load on RTX 4090 (24 GB): ~19.5 GB with cache_size=131072, cache_mode=Q8, max_seq_len=131072.

Build provenance

• —Quantizer: turboderp-org/exllamav3 (tag v1.4.2)
• —Host: Windows 11, Python 3.14.7, torch 2.11.0+cu128, CUDA 12.8, triton-windows 3.7.1
• —GPU: NVIDIA GeForce RTX 4090 (24 GB, sm 8.9), driver 610.88
• —Build wall time: 1 h 21 min 38 s end-to-end (per-module sum: 1 h 43 min)
• —Modules quantized: 70 (1 embedtokens + 64 transformer + 1 norm + 1 lmhead + 3 mtp/vision/projection)
• —Calibration source: exllamav3 v1.4.2 standardcaldata (6 files, ~5.6 MB)

License

Apache License 2.0. See `LICENSE` — inherited from the upstream Blackfrost repo, which traces to Alibaba Cloud (Qwen authors).