Modelpublic

distil-labs/distil-qwen3-1.7b-customer-support-deferral-gguf

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes49downloads

Model Card

Distil-Qwen3-1.7B-Customer-Support-Deferral (GGUF)

GGUF build of distil-labs/distil-qwen3-1.7b-customer-support-deferral, for serving with llama.cpp.

A fine-tuned Qwen3-1.7B model for multi-turn airline customer support that runs as the small tier of a two-model cascade: it handles most support turns itself and defers genuinely-hard turns to a larger model by emitting a defer_to_larger_model tool call. Every assistant action is a single tool call, including talking to the customer via respond_to_user, so a thin orchestrator can drive it.

Results

Evaluated on a held-out set of airline customer-support turns, scored by an independent GLM-5 judge (score = fraction of responses rated correct).

System	Quality	Frontier-model calls
Frontier model alone (GLM-5)	0.80	100%
This model + escalation (local)	~0.75	~4%
Untrained Qwen3-1.7B	0.42	0%

Fine-tuning lifts the local 1.7B from 0.42 to ~0.75 (closing roughly 85% of the gap to its frontier-scale teacher), while running ~96% of turns locally and escalating only the hardest ~4% to the larger model. The escalation is a cost/safety mechanism, not a quality boost over the small model alone.

Usage (llama.cpp)

bash

hf download distil-labs/distil-qwen3-1.7b-customer-support-deferral-gguf \
    distil-qwen3-1.7b-customer-support-deferral.gguf --local-dir models

llama-server \
    --model models/distil-qwen3-1.7b-customer-support-deferral.gguf \
    --port 8000 \
    --jinja

Then query the OpenAI-compatible API at http://127.0.0.1:8000/v1. The airline policy (system prompt) and the 16 tool schemas ship with the demo app as job_description.json.

Demo App

This model powers the Flexible Customer Support Bot demo, a terminal cascade where this local SLM handles most airline-support turns and defers hard turns to a larger, OpenAI-compatible model.

File

File	Format	Notes
`distil-qwen3-1.7b-customer-support-deferral.gguf`	BF16	Full-precision build (16 bpw). Quantize with `llama-quantize` if you need a smaller footprint.

License