rihuwa/Reina-Magistral-GGUF

Available Quantizations

Training Details

• —LoRA Rank: 128
• —LoRA Alpha: 64
• —Target Modules: All attention and FFN layers
• —Training Format: Mistral v7 Tekken chat template
• —Sequence Length: 2048 tokens
• —Base Precision: BF16

Usage

With llama.cpp

# Download model (requires HF authentication for private repo)
huggingface-cli login
huggingface-cli download rihuwa/Reina_Magitral_gguf magistral-small-2509-text-only-Q5_K_M.gguf --local-dir ./models

# Run inference
./llama-cli -m ./models/magistral-small-2509-text-only-Q5_K_M.gguf -p "Your prompt here" -n 256 \
  --temp 0.7 --top-p 0.95

With Ollama

# Download the GGUF file first
huggingface-cli login
huggingface-cli download rihuwa/Reina_Magitral_gguf magistral-small-2509-text-only-Q5_K_M.gguf --local-dir ./models

# Create Modelfile (the template uses Mistral v7 format)
cat > Modelfile << 'EOF'
FROM ./models/magistral-small-2509-text-only-Q5_K_M.gguf

PARAMETER temperature 0.7
PARAMETER top_p 0.95
PARAMETER num_ctx 8192
EOF

# Create and run model
ollama create reina-magistral -f Modelfile
ollama run reina-magistral

Important Notes for Magistral Models

1. 1.Special Tokens: Magistral uses [THINK] and [/THINK] tags for reasoning
2. 2.Context Window: 128k tokens (optimal performance under 40k)
3. 3.Chat Format: Mistral v7 Tekken template
4. 4.Recommended Settings:
5. 5.Temperature: 0.7
6. 6.Top-p: 0.95
7. 7.Top-k: 40

Model Architecture

• —Type: Mistral-based (24B parameters)
• —Tokenizer: Tekken tokenizer with special reasoning tokens
• —Vocabulary: Extended for reasoning capabilities
• —Attention: Sliding window + global attention

Quantization Details

Both quantizations preserve the quality of the LoRA fine-tuning:

• —Q8_0: 8-bit quantization, virtually no quality loss
• —Q5_K_M: 5-bit with K-quant method, excellent quality/size balance

Performance

Tested on NVIDIA GPUs:

• —Q8_0: Requires ~28GB VRAM
• —Q5_K_M: Requires ~18GB VRAM (fits on single 24GB GPU)

Related Repositories

• —LoRA Adapter: rihuwa/Reina_Magistral_LoRA
• —Base Model: Darkhn/Magistral-Small-2509-Text-Only

License

This model inherits the Apache 2.0 license from the base Magistral model.

Last Updated: 2025-11-23