ogiwrghs/Phi-3-medium-128k-instruct-GGUF

Phi-3 Medium 128k Instruct (GGUF)

This repository provides GGUF quantized models for Microsoft Phi-3 Medium 128k Instruct.

Quantization was performed using llama.cpp with Importance Matrix (IQ) techniques to optimize VRAM usage, inference speed, and model quality.

📊 Benchmarks (NVIDIA T4)

Benchmarks were collected on Google Colab (Tesla T4) using llama-bench and llama-perplexity.

Notes

• —IQ4_XS delivers ~87% higher throughput than Q5KM with negligible quality loss.
• —2-bit quantization causes severe degradation on Phi-3 Medium.

📦 Available Files

RAM estimates include KV-cache overhead.

💡 Quantization Info

Importance Matrix (IQ) quantization uses calibration data to preserve the most important weights, significantly reducing quality loss compared to standard K-quant methods at similar sizes.

Recommended: IQ4_XS Fits in ~8GB VRAM and sustains 20+ tokens/sec on consumer GPUs (RTX 3060 / 3070 / 4060).

🚀 Usage

⚠️ Choose ONE of the following options. Do NOT run both.

Option 1: CLI (llama.cpp)

Use this if you want an interactive terminal session.

./llama-cli \
  -m Phi-3-medium-128k-instruct-IQ4_XS.gguf \
  -n -1 \
  --color \
  -cnv

Option 2: Python (llama-cpp-python)

from llama_cpp import Llama

# Set gpu_layers to -1 to offload all layers to GPU
llm = Llama(
    model_path="./Phi-3-medium-128k-instruct-IQ4_XS.gguf",
    n_ctx=4096, 
    n_gpu_layers=-1,
    verbose=True
)

output = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum physics."},
    ]
)

print(output['choices'][0]['message']['content'])