CoolFace
Modelpublic

XythicK/Plano-Orchestrator-30B-A3B-GGUF

sourceHugging Faceupdated 9mo agoView on Hugging Face
1likes455downloads
Model Card

Plano-Orchestrator-30B-A3B-GGUF

🧠 Model Overview

Plano-Orchestrator-30B-A3B-GGUF is a quantized version of Plano-Orchestrator-30B-A3B-GGUF, optimized for efficient inference with reduced memory usage and faster runtime while preserving as much of the original model quality as possible.

This repository provides multiple quantized variants suitable for:

  • —Local inference
  • —Low-VRAM GPUs
  • —CPU-only environments

🔗 Original Model

  • —Base model: Plano-Orchestrator-30B-A3B
  • —Original repository: https://huggingface.co/katanemo/Plano-Orchestrator-30B-A3B
  • —Original authors: Katanemo

📦 Quantization Details

  • —Quantization method: GGUF
  • —Quantization tool: llama.cpp
  • —Precision: Mixed (2-8,bit depands in variant)
  • —Activation aware: No (weight-only quantinization)
  • —Group size: 256 (K-quant variants)

📦 Available Quantized Files

Quant FormatFile NameApprox. SizeVRAM / RAM NeededNotes
Q2_Kplano-orchestrator-q2_k.gguf~11.3 GB~14 GBExtreme compression; noticeable quality loss
Q3KSplano-orchestrator-q3_k_s.gguf~13.3 GB~15.4 GBSmaller, faster, lower quality
Q3KMplano-orchestrator-q3_k_m.gguf~14.7 GB~16 GBBetter balance than Q3KS
Q3KLplano-orchestrator-q3_k_l.gguf~15.9 GB~18 GBHighest-quality 3-bit variant
Q4_0plano-orchestrator-q4_0.gguf~17.3 GB~19.3 GBLegacy format; simpler quantization
Q4KSplano-orchestrator-q4_k_s.gguf~17.5 GB~19.5 GBSmaller grouped 4-bit
Q4KMplano-orchestrator-q4_k_m.gguf~ GB~ GBRecommended default
Q5_0plano-orchestratorq-5_0.gguf~21 GB~23 GBHigher quality, larger size
Q5KSplano-orchestrator-q5_k_s.gguf~21.1 GB~23 GBEfficient high-quality variant
Q5KMplano-orchestratorq-5_K_M.gguf~X GB~ GBNear-FP16 quality
Q6_Kplano-orchestratorq-6_k.gguf~25.1 GB~27 GBMinimal quantization loss
Q8_0plano-orchestrator-q8_0.gguf~32.5 GB~36 GBMaximum quality; large memory
💡 Recommendation: Start with Q4_K_M for the best quality-to-performance ratio.

🚀 Usage Example

llama.cpp

bash
./main -m plano-orchestrator-30b-a3b-q6_k.gguf -p "Your prompt here" -n 256

Python (llama-cpp-python)

Python
from llama_cpp import Llama

llm = Llama(
    model_path="plano-orchestrator-30b-a3b-q6_k.gguf",
    n_ctx=4096,
    n_threads=8
)

print(llm("Your prompt here"))

🙋 Contact

Maintainer: M Mashhudur Rahim [XythicK]

Role: Independent Machine Learning Researcher & Model Infrastructure Maintainer

(Focused on model quantization, optimization, and efficient deployment)

For issues, improvement requests, or additional quantization formats, please use the Hugging Face Discussions or Issues tab.

❤️ Acknowledgements

Thanks to the original model authors for their ongoing contributions to open AI research, and to Hugging Face and the open-source machine learning community for providing the tools and platforms that make efficient model sharing and deployment possible.