localslm/Qwen3.5-9B-MTP-Uncensored

Uncensored Qwen3.5-9B-MTP GGUF

Example Hybrid Setup: CPU (LLM) + GPU (Vision, ~4GB VRAM)

This setup launches a highly optimized hybrid CPU/GPU vision-enabled server using llama.cpp, specifically tuned for maximum performance, minimal memory footprint, and high-quality generation on budget hardware:

• —Fast CPU Text Generation: Leverages Multi-Token Prediction (MTP) speculative decoding. The process is pinned to physical CPU cores, dramatically increasing cache hits and reducing latency.
• —Instant Image Preprocessing (~1 second): Offloads the multimodal projector to even a low-end GPU (4GB VRAM). Even with a massive budget of 4,096 visual tokens, image processing takes under a second on GPU.
• —Massive 49K Context Window: Achieved within standard RAM limits by utilizing 4-bit KV-cache quantization.
• —Advanced Repetition Control: Uses DRY (Don't Repeat Yourself) sampling instead of static penalties to ensure natural, loop-free text generation.
• —WebUI & API: Features an OpenAI-compatible local API/WebUI server on port 8080.

llama-server.exe ^
    -m "D:\LLMs\Qwen3.5-9B-MTP-Uncensored\model.gguf" ^
    --mmproj "D:\LLMs\Qwen3.5-9B-MTP-Uncensored\mmproj.gguf" ^
    --seed -1 ^
    -np 1 ^
    --image-min-tokens 4096 ^
    --image-max-tokens 4096 ^
    -t 6 --threads-batch 6 ^
    -Cr 0-5 --cpu-strict 1 ^
    --prio 2 --poll 100 ^
    -b 4224 -ub 2048 --mtmd-batch-max-tokens 2048 ^
    -ctk q4_0 -ctv q4_0 ^
    -ctkd q4_0 -ctvd q4_0 ^
    --cache-reuse 256 ^
    --cache-prompt ^
    --slot-prompt-similarity 0.0 ^
    --swa-full ^
    -lm mlock ^
    -ngl 0 ^
    --mmproj-offload ^
    -fa on ^
    -c 49152 ^
    --temp 0.3 ^
    --min-p 0.05 ^
    --top-p 0.95 ^
    --top-k 40 ^
    --dry-multiplier 0.8 ^
    --dry-base 1.75 ^
    --dry-allowed-length 2 ^
    --dry-penalty-last-n 256 ^
    --presence-penalty 0.0 ^
    --repeat-penalty 1.0 ^
    --reasoning off ^
    --jinja ^
    --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.20 ^
    --metrics^
    --host 127.0.0.1 --port 8080

This command launches `llama-server.exe` (the server binary provided by llama.cpp) to host a local REST API / web interface for a vision-enabled Qwen3.5-9B on Windows:

1. Model & Multimodal Files

• —`-m` & `--mmproj` Paths to the main language model and the multimodal projector (vision encoder).
• —`--image-min-tokens 4096` & `--image-max-tokens 4096` Forces image processing to use a fixed allocation of 4,096 visual tokens per image input, ensuring maximum visual detail and OCR precision.
• —`--mmproj-offload` Crucial optimization: Offloads the heavy vision encoder to the GPU. Even on a cheap GPU, processing 4,096 visual tokens completes in < 1 second.

2. CPU Threading & Hardware Optimization

• —`-t 6` & `--threads-batch 6` Allocates 6 CPU threads for token generation and batch processing.
• —`-Cr 0-5 --cpu-strict 1` Pins the process strictly to physical CPU cores 0 through 5. This prevents the OS from migrating threads across cores, vastly improving CPU cache hit rates and generation speed.
• —`--prio 2` & `--poll 100` Sets the server to high process priority (2) and lowers the polling interval (100 ms) for faster API responsiveness.
• —`-lm mlock` Locks model memory directly in physical RAM, preventing the OS from swapping it to the disk/pagefile (replaces older --no-mmap arguments).
• —`-ngl 0` Forces the main LLM to run entirely on the CPU.
• —`-fa on` Enables FlashAttention to significantly reduce RAM usage during context processing.

3. Batching, Context, & KV-Cache (Memory Savings)

• —`-c 49152` & `-np 1` Sets a massive 49,152 token context window, optimized for a single user/slot (-np 1).
• —`-b 4224` & `-ub 2048` Sets the prompt batch size to 4224 and micro-batch size to 2048, balancing RAM spikes and CPU load.
• —`--mtmd-batch-max-tokens 2048` Limits multimodal chunking to 2048 tokens, preventing Out-Of-Memory (OOM) errors during heavy image ingestion.
• —`-ctk q4_0 -ctv q4_0` & `-ctkd q4_0 -ctvd q4_0` Applies 4-bit quantization to the KV-cache (for both main and draft models). This is what makes the 49K context window possible on standard consumer RAM setups without degrading output quality.
• —`--cache-prompt`, `--cache-reuse 256`, `--slot-prompt-similarity 0.0` Enables aggressive prompt caching. This skips reprocessing the system prompt and chat history, drastically speeding up multi-turn conversations.
• —`--swa-full` Enables full Sliding Window Attention, further optimizing memory footprint.

4. Sampling & DRY Repetition Control

• —`--temp 0.3`, `--top-p 0.95`, `--top-k 40`, `--min-p 0.05` A precise, low-temperature sampling setup that relies heavily on Min-P to truncate low-probability garbage tokens, leading to highly coherent and logical responses.
• —`--seed -1` Ensures a random seed for generation on every prompt.
• —DRY (Don't Repeat Yourself) Sampler (Replaces standard repetition penalties):
• —`--dry-multiplier 0.8`, `--dry-base 1.75`, `--dry-allowed-length 2`, `--dry-penalty-last-n 256` Instead of statically penalizing words (which breaks formatting/code), DRY dynamically detects sequences of repeating tokens and exponentially penalizes them, effectively curing model loops.
• —`--presence-penalty 0.0` & `--repeat-penalty 1.0` Standard penalties are disabled to allow the DRY sampler to work without interference.

5. Speculative Decoding & Template Settings

• —`--spec-type draft-mtp` Enables Multi-Token Prediction (MTP).
• —`--spec-draft-n-max 2` & `--spec-draft-p-min 0.20` Generates up to 2 draft tokens per step, but only if the model is at least 20% confident (0.20). This prevents wasting CPU cycles on bad guesses, resulting in a smoother, faster generation rate.
• —`--reasoning off` Natively disables the DeepSeek/Qwen thinking/reasoning steps, guaranteeing immediate final-answer generation and saving compute time.
• —`--jinja` & `--metrics` Enables Jinja2 parsing for complex chat templates and exposes an endpoint for performance monitoring.