Abiray/Mistral-Medium-3.5-128B-Q4_K_M-GGUF

Mistral-Medium-3.5-128B - GGUF (Q4KM)

🔄 Changelog / Updates

• —Config Bug Fix: A bug was recently discovered and fixed in the original model's Transformers config, which propagated to the initial GGUF release. This repository has now been updated with the newly fixed weights/config.
• —Impact: Users will experience better, more stable performance.
• —Action Required: If you are using the older version, it is strongly recommended that you pull the latest files from this repository. Thanks to the community for the heads-up!

This repository contains the Q4_K_M GGUF quantized version of Mistral-Medium-3.5-128B.

• —Base Model: mistralai/Mistral-Medium-3.5-128B
• —Quantization Level: Q4KM (Good quality, recommended balance of size and performance).

💻 Hardware Requirements

• —Total File Size: ~74.9 GB
• —Memory Required: Minimum ~85 GB of total combined memory (RAM + VRAM) to load the model and maintain a usable context window.
• —Recommended Setup: Apple Silicon (M-series with 96GB/128GB Unified Memory) or multi-GPU rigs (e.g., 4x 3090/4090) for full VRAM offloading.

📝 Prompt Format

This model requires the standard Mistral instruction format to function correctly and avoid hallucination or broken outputs.

<s>[SYSTEM_PROMPT]{system_prompt}[/SYSTEM_PROMPT][MODEL_SETTINGS]{"reasoning_effort": "none"}[/MODEL_SETTINGS][INST]{prompt}[/INST]