Modelpublic

tinyopsec/Qwen3.5-4B-Unredacted-MAX-GGUF

sourceHugging Faceapache-2.0updated 9d agoView on Hugging Face

0likes2.1kdownloads

Model Card

Qwen3.5-4B-Unredacted-MAX - GGUF

GGUF quantized versions of prithivMLmods/Qwen3.5-4B-Unredacted-MAX, a 4.5 billion parameter language model based on the Qwen3.5-4B architecture, optimized for improved loading stability and compatibility with modern Transformers pipelines.

Model Details

—Base Model: Qwen/Qwen3.5-4B
—Fine-tuned by: prithivMLmods
—Architecture: Qwen2 (28 layers, 28 attention heads)
—Context Length: 32768 tokens
—Vocabulary Size: 151936
—Parameters: 4.5B

Quantization

Filename	Bits	Size	Use Case
`model_f16.gguf`	16	~8.4 GB	Maximum quality, high VRAM requirement
`model_q8_0.gguf`	8	~4.5 GB	High quality, moderate VRAM
`model_q6_k.gguf`	6	~3.4 GB	Good quality, balanced VRAM
`model_q5_k_m.gguf`	5	~2.8 GB	Recommended for most use cases
`model_q5_k_s.gguf`	5	~2.5 GB	Compact, minimal quality loss
`model_q4_k_m.gguf`	4	~2.1 GB	Good balance, low VRAM
`model_q4_k_s.gguf`	4	~1.9 GB	Compact model
`model_q3_k_l.gguf`	3	~1.7 GB	Very compact
`model_q3_k_m.gguf`	3	~1.5 GB	Minimal size
`model_q3_k_s.gguf`	3	~1.4 GB	Extreme compression
`model_q2_k.gguf`	2	~1.1 GB	Maximum compression

VRAM Requirements

Quantization	VRAM	Recommended Device
F16	9 GB	High-end GPU
Q8_0	5 GB	Mid-range GPU
Q6_K	3.5 GB	Mid-range GPU
Q5KM	3 GB	Standard GPU
Q4KM	2.5 GB	Standard GPU / Laptop GPU
Q3KM	2 GB	Laptop GPU / CPU
Q2_K	1.5 GB	CPU with sufficient RAM

Usage

llama.cpp

bash

./main -m model_q5_k_m.gguf -n 256 -p "You are a helpful assistant."

llama-cpp-python

python

from llama_cpp import Llama

llm = Llama(
    model_path="model_q5_k_m.gguf",
    n_ctx=32768,
    n_threads=8
)

response = llm("What is machine learning?", max_tokens=256)
print(response["choices"][0]["text"])

LM Studio

Download the quantized GGUF file and load it in LM Studio's model picker.

Ollama

bash

ollama pull tinyopsec/qwen3.5-4b-unredacted-max-gguf:q5_k_m
ollama run tinyopsec/qwen3.5-4b-unredacted-max-gguf:q5_k_m

Model Capabilities

This model excels at:

—Instruction Following: Optimized for direct response generation and multi-step instructions
—Reasoning Tasks: Capable of complex reasoning and problem-solving
—Text Generation: High-quality content generation across diverse topics
—Research & Experimentation: Designed for transformer behavior analysis and instruction dynamics understanding
—Lightweight Deployment: Efficient 4.5B parameter architecture suitable for local inference

License

This quantized version maintains the original model's Apache 2.0 license. See the original model card for full details.

Disclaimer

These are community quantizations. For official support and documentation, refer to the original model repository and llama.cpp documentation.