andrevp/Qwen3.5-9B-Distilled-OPUS-Heretic-MLX-VLM-4bit

Qwen3.5-9B Distilled OPUS Heretic - MLX-VLM 4bit

4-bit quantized MLX-VLM conversion of an abliterated Qwen3.5-9B model distilled from Claude Opus 4.6 reasoning, optimized for Apple Silicon.

Size: ~5.6 GB | Bits/weight: 5.059 | Quality: Reduced; fastest and smallest variant

Background

This model starts from Jackrong/Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled, a Qwen3.5-9B base fine-tuned via knowledge distillation from Claude Opus 4.6 to replicate its chain-of-thought reasoning style.

Abliteration was applied using the technique from Arditi et al. (2024), adapted with a custom script to handle the hybrid DeltaNet/full-attention architecture. The result is a model that retains strong reasoning and vision capabilities while removing refusal behavior.

The model was then converted to MLX-VLM format and quantized to 4-bit for Apple Silicon inference. This variant offers the smallest footprint at some cost to output quality; prefer the 8-bit or fp16 variants where memory allows.

Architecture

• —Type: Qwen3_5ForConditionalGeneration (multimodal)
• —Layers: 32 total — 24 linear attention (DeltaNet) + 8 full attention
• —Hidden size: 4096 | Intermediate size: 12288
• —Vision encoder: 27-layer ViT
• —Inputs: Text, images, video

Confirmed Capabilities

• —Vision: Correctly describes image content
• —Reasoning: Step-by-step mathematical problem solving (e.g., integration by parts)
• —Uncensored: Responds to sensitive prompts without refusal

Usage

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model_path = "andrevp/Qwen3.5-9B-Distilled-OPUS-Heretic-MLX-VLM-4bit"
model, processor = load(model_path)
config = load_config(model_path)

# Text-only
prompt = apply_chat_template(processor, config, "Your question here", num_images=0)
result = generate(model, processor, prompt, max_tokens=500)
print(result.text)

# Vision
prompt = apply_chat_template(processor, config, "Describe this image", num_images=1)
result = generate(model, processor, prompt, max_tokens=500, image=["image.jpg"])
print(result.text)

Model Family

Credits

• —Base distillation: Jackrong/Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled
• —Abliteration technique: Arditi et al., "Refusal in Language Models Is Mediated by a Single Direction" (2024)
• —MLX-VLM framework: Apple MLX-VLM