uqer1244/Qwen3.8-27B-GSQ-RCO-MLX

Qwen3.8-27B-GSQ-RCO-MLX

An MLX derivative of `ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF`, with its GSQ-RCO mixed-IQ weights packaged as a lossless packed Safetensors file for inference on Apple Silicon with MLX and custom Metal kernels.

This is not a standard mlx-lm quantized checkpoint. The tensors in model.safetensors are uint8 containers holding the original GGUF packed payloads. Use the separate mlx_gsq runtime; loading this file directly with mlx_lm.load() will not work.

Contents

• —model.safetensors: 866 tensors and the original packed quantized payloads
• —model.safetensors.manifest.json: shape and GGUF quantization metadata
• —tokenizer and Qwen3.8 configuration files at repository root

The runtime supports all 11 quantized tensor types used by this checkpoint: IQ1_M, IQ1_S, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, Q2_K, Q4_K, and Q6_K. Plain F32 and BF16 tensors are also loaded without conversion.

Requirements

• —Apple Silicon Mac with Metal support
• —Python 3.10 or newer
• —MLX and mlx-lm
• —approximately 10.4 GB of unified memory for a minimal inference run; additional headroom is recommended

Installation

Install the runtime from GitHub. The first run downloads the model from the Hugging Face Hub automatically:

python -m pip install \
  'mlx-gsq-rco[mlx,integration] @ git+https://github.com/uqer1244/mlx-GSQ-RCO.git'

Runtime development and issue tracking: `uqer1244/mlx-GSQ-RCO`

Generation

mlx-gsq-generate \
  uqer1244/Qwen3.8-27B-GSQ-RCO-MLX \
  --prompt "Hello" \
  --max-tokens 32 \
  --verbose

Python usage:

from mlx_gsq import load
from mlx_lm.generate import generate

model, tokenizer = load("uqer1244/Qwen3.8-27B-GSQ-RCO-MLX")
text = generate(model, tokenizer, prompt="Hello", max_tokens=32)
print(text)

Format

The Safetensors metadata declares:

format=mlx-gsq-packed-v1
architecture=qwen35
source_format=GGUF

Each packed tensor is stored byte-for-byte as uint8. Logical shape, quantization type, block size, and byte count are recorded both in embedded Safetensors metadata and in the sidecar manifest. The custom runtime routes each matrix to the correct Metal kernel at execution time.

Current limitations

• —Prefill currently uses the functional multi-row QMV path. A tiled QMM kernel is still planned for higher prompt-processing throughput.
• —This repository requires the separate custom runtime and cannot be loaded as an ordinary MLX-LM checkpoint.
• —The included runtime currently targets this Qwen3.8/Qwen3.5 hybrid architecture and its tensor naming scheme.

Provenance and license

• —Original base model: Qwen/Qwen3.8-27B
• —Direct parent/quantized source: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
• —Packed Safetensors conversion preserves the GGUF tensor payloads; it does not requantize the model.

The model card declares the upstream Apache-2.0 license. Review and retain all upstream license notices and usage conditions when redistributing the model.