cspenn/Skyfall-31B-v4.2-MLX-Mixed-4-8

Skyfall-31B-v4.2 MLX Mixed 4/8-bit

An MLX mixed-precision quantization of TheDrummer/Skyfall-31B-v4.2 for inference on Apple Silicon using the mlx-lm framework.

Model overview

Skyfall-31B-v4.2 is a 31-billion parameter dense language model fine-tuned by TheDrummer for creative writing and roleplay. It is built on the Mistral-Small-3.2-24B-Instruct base, upscaled to 31B parameters, and trained to excel at storytelling, character interpretation, and creative dialogue. It prioritizes expressive generation quality over factual benchmark performance.

Quantization details

This is a mixed 4/8-bit quantization using the MLX affine scheme (group_size: 64). Layers that are most sensitive to precision loss (embeddings, attention projections, early and final layers) are preserved at 8-bit; the majority of the feed-forward weights are compressed to 4-bit.

This is a higher-fidelity profile than a uniform 4-bit quant — the 8-bit-heavy mix preserves more of the creative generation quality that makes Skyfall distinctive.

Disk size: ~23 GB across 5 safetensors shards.

Why v2? Known issues in v1 and their fixes

This is the second quantization pass (-v2). Two bugs in the first attempt caused incoherent or repetitive output:

1. RoPE theta misconfiguration (critical — caused incoherent output)

transformers 5.0 writes rope_theta only inside a nested rope_parameters block in config.json. mlx_lm's llama.py (which handles model_type: mistral) declares rope_theta: float = 10000 at the top level and has no rope_parameters field, so from_dict silently discarded it. The model ran with θ=10,000 instead of the correct θ=1,000,000,000 — effectively no long-context positional encoding. Output was incoherent for anything beyond trivially short prompts.

Fix applied: rope_theta is hoisted from the nested block to the top level in config.json. The weights are unchanged; only the config layout was corrected. You can verify: jq .rope_theta config.json should return 1000000000.

2. Attention projection sensitivity (caused degenerate repetition)

With Grouped Query Attention (GQA) and only 8 KV heads, the k_proj and v_proj matrices are small (5120→1024). At 4-bit quantization, the error in these small tensors is disproportionately large relative to their contribution to attention. At standard sampling (temp=1.0, no repetition penalty) this caused persistent repetition loops. Raising all attention projections (q_proj, k_proj, v_proj, o_proj) to 8-bit eliminated the issue.

The 8-bit-heavy profile (276 layers at 8-bit) reflects these sensitivity findings.

Hardware requirements

This model requires Apple Silicon with the MLX framework.

At 32 GB you'll comfortably run 32K+ context; the full 128K window needs additional headroom.

Usage

Install dependencies:

pip install mlx-lm

CLI:

mlx_lm.generate \
  --model cspenn/Skyfall-31B-v4.2-MLX-Mixed-4-8 \
  --prompt "Continue this story: The lighthouse keeper had not seen another soul in three years, until the night a boat washed ashore with no one aboard." \
  --max-tokens 512 \
  --temp 0.8

Python:

from mlx_lm import load, generate

model, tokenizer = load("cspenn/Skyfall-31B-v4.2-MLX-Mixed-4-8")

prompt = (
    "Continue this story: The lighthouse keeper had not seen another soul "
    "in three years, until the night a boat washed ashore with no one aboard."
)

# Apply the model's chat template
messages = [{"role": "user", "content": prompt}]
formatted = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

response = generate(
    model, tokenizer, prompt=formatted, max_tokens=512, temp=0.8, verbose=True
)

Tip: For creative writing, temperatures in the range of 0.7–1.0 give more varied and expressive output. For structured tasks, lower to 0.3–0.5.

Lineage

mistralai/Mistral-Small-3.2-24B-Instruct  →  TheDrummer/Skyfall-31B-v4.2 (upscaled + fine-tuned)  →  this MLX mixed-4/8 quant

License

Apache 2.0 — inherited from the Mistral-Small base. See TheDrummer/Skyfall-31B-v4.2 for the upstream model terms.