Modelpublic

Yanun/Swift-Qwen3.8-27b-oQ4e-mtp

sourceHugging Faceotherupdated 9d agoView on Hugging Face

7likes1.3kdownloads

Model Card

Swift-Qwen3.8-27b-oQ4e-mtp

Model architecture

A 27B-class dense multimodal model stored in MLX format. It contains a text backbone, a vision encoder, and one multi-token prediction (MTP) layer.

Component	Structure
Text backbone	64 layers; hidden size 5,120; feed-forward size 17,408
Attention layout	48 linear-attention layers and 16 full-attention layers, with full attention every fourth layer
Full attention	24 query heads, 4 key/value heads, head dimension 256
Vocabulary	248,320 tokens
Configured context limit	262,144 tokens; usable length depends on runtime settings and available memory
Vision encoder	27 layers; hidden size 1,152; 16 attention heads; 16 × 16 image patches
Vision-to-text connection	Vision features are projected to the text hidden size of 5,120
MTP	One additional prediction layer with attention and feed-forward projections

Weight precision

The oQ4e checkpoint uses mixed precision rather than uniform 4-bit weights:

—The default quantization is 4-bit affine, with 64 values per group.
—187 modules have explicit 5-bit overrides in config.json.
—The MTP layer's seven large attention and feed-forward matrices use 4-bit weights.
—The MTP fusion matrix (mtp.fc) and normalization weights remain BF16. The fusion matrix maps 10,240 input features to 5,120 output features.
—The MTP quantization scales and offsets are stored in BF16.

This checkpoint retains BF16 floating-point weights in the MTP fusion and normalization layers alongside its quantized matrices. Exact per-module settings are recorded in config.json.

Source and license

Source revision: 54e66d6c81439bd4fda5ef9a690fa571e3b0d272.

Original model by UkisAI. This conversion does not change the upstream Swift Open License v1.0 terms. Consult the source model license and access information.