iromu/Qwen3-0.6B-tools-GGUF

Qwen3-0.6B-tools GGUF

The Qwen3-0.6B tool-calling fine-tune in GGUF format, for llama.cpp-style runtimes. The model was supervised-fine-tuned with LoRA so it reliably emits OpenAI-style tool calls for agentic workflows. The full BF16 model and its validation matrix live at iromu/Qwen3-0.6B-tools.

Base model

This model was fine-tuned from:

Qwen/Qwen3-0.6B

GGUF files

The model is provided in GGUF format at the following precisions:

On this model the exact-args fidelity (measured on the sft_tools validation split; full numbers in the serving matrix below) is not a smooth ramp in quant size. Q4KM drops to about a third of the BF16 rate; Q5KM and Q6K sit in the same ~54% band, so Q6K costs more bytes for no fidelity gain over Q5KM; and Q8_0 is the fidelity pick — it reaches parity with the BF16 conversion (~66% exact args) and is the quant to use when the exact tool arguments matter.

Training

Supervised fine-tuning with LoRA, run with NeMo-RL, assistant-turn-masked cross-entropy over the tool-calling logs.

• —LoRA: dim 32, alpha 32, dropout 0.05, target modules *.proj (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
• —Max sequence length: 4096
• —Precision: bfloat16
• —Optimizer: AdamW, lr 5e-5, weight decay 0.01
• —Schedule: 15-step linear warmup, then cosine decay to 1e-6
• —Global batch size: 64 (micro batch 2)
• —Training steps: 336 (validation every 84)
• —Loss masked to assistant turns via the chat template's generation keyword

Dataset

Training used the sft_tools split of the r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation dataset (OpenAI messages format; assistant turns carry tool_calls).

Intended use

• —Structured tool / function calling in llama.cpp-style runtimes
• —Agent-style multi-step interactions
• —Small-footprint on-device or edge deployment

It is not intended to be a general replacement for larger Qwen models.

Sibling artifacts

• —BF16 model + validation matrix: iromu/Qwen3-0.6B-tools
• —NVFP4 checkpoint: iromu/Qwen3-0.6B-tools-NVFP4

Usage

Load a GGUF quant with llama.cpp:

llama-cli -hf iromu/Qwen3-0.6B-tools-GGUF:Q4_K_M

Or with the llama.cpp Python bindings:

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_path="iromu/Qwen3-0.6B-tools-GGUF",
    filename="Qwen3-0.6B-tools-Q5_K_M.gguf",
    n_gpu_layers=99,
)

<!-- VALIDATION:BEGIN (auto-generated, do not edit) -->

Serving matrix

Quantizations of iromu/Qwen3-0.6B-tools, measured on the sft_tools held-out validation split, greedy decoding, via llama.cpp. The full BF16 model and its training ladder live at iromu/Qwen3-0.6B-tools.

BASE is the SFT model iromu/Qwen3-0.6B-tools itself, measured at cap 1024 (the quants at cap 384); no quant truncated at its cap except 7 rows each for Q4KM and Q5KM, so the gap is model fidelity, not a truncation artifact. Q8_0 is the fidelity pick — it loses the least vs BASE (−5.1pp). <!-- VALIDATION:END -->