ShahzebKhoso/Qwen3-4B-SafeRL-GGUF

Qwen3-4B-SafeRL - GGUF Quantized Versions

This repository provides GGUF quantized versions of Qwen/Qwen3-4B-SafeRL, converted with llama.cpp.

The base model was first exported from Hugging Face format to GGUF (FP16) and then quantized into multiple formats. These variants offer different trade-offs between model size, inference speed, and output quality.

🔧 Model Details

• —Base model: Qwen/Qwen3-4B-SafeRL
• —Architecture: Qwen 3 (4B parameters)
• —Format: GGUF
• —Intended use: Safe RL research & alignment tasks
• —Conversion tool: convert_hf_to_gguf.py (from llama.cpp)
• —Quantization tool: llama-quantize

📊 Quantized Versions

🚀 Usage

🖥️ llama.cpp

./main -m Qwen3-4B-SafeRL-Q4_K_M.gguf -p "Hello, SafeRL!"

🐍 Python

from huggingface_hub import hf_hub_download
from llama_cpp import Llama

model_path = hf_hub_download(
    repo_id="YOUR_USERNAME/Qwen3-4B-SafeRL-GGUF",
    filename="Qwen3-4B-SafeRL-Q4_K_M.gguf"
)

llm = Llama(model_path=model_path)

output = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a safe RL assistant."},
        {"role": "user", "content": "Hello, SafeRL!"}
    ],
    max_tokens=100
)

print(output["choices"][0]["message"]["content"])

These GGUF versions are optimized for fast inference with CPU/GPU runtimes like llama.cpp, Ollama, and LM Studio.