minhlt12/qwen3.5-4b-lora-vi-ticket-triage

LoRA adapter — Vietnamese customer-support ticket triage (Qwen3.5-4B)

LoRA adapter fine-tuned to turn a Vietnamese customer-support ticket into a strict 4-field JSON triage object: intent, urgency, product, sentiment.

Trained as coursework for AICB-P2T3 Day 21 — Fine-tuning & Safety (Track 3).

## ⚠️ Read this before using the adapter This adapter fails its own regression gate and is published as a teaching artifact, not as a production model. It improves the target task (0.765 → 0.970) but degrades unrelated general-knowledge ability by 0.302 (0.758 → 0.456), which is over 15× the lab's 0.020 tolerance. This is textbook catastrophic forgetting: 100% of the training data was ticket → JSON, with no general-domain replay mixed in. Use it to reproduce the measurement, not to serve traffic. If you want to fix it, mix 1–5% general instruction data into the training set and re-run.

Results

Measured on a frozen 50-item target set and a 15-item general-knowledge regression set. All three rows use the same eval harness; (a) and (b) share the base weights and differ only in prompt.

Verdict: FAILED — target Δ +0.205, regression Δ −0.302 (tolerance 0.020).

Baseline (a) scores 0.000 on target because it scores 0.00 on format: it rarely emits parseable JSON at all, so no field can be scored. The failure is one of output format, not of classification.

Known failure mode

The adapter gets 44/50 target items fully correct. All 6 errors are the same error: urgency predicted as trung_binh when the gold label is thap, on tickets containing the phrase "Khi nào tiện" ("whenever it's convenient").

This is not a data-coverage gap — Khi nào tiện appears 30 times in training, all labelled thap, more often than Hỏi cho biết thôi (22) which the adapter learns perfectly. The likely cause is a conflict with the base model's prior: the phrase opens with Khi nào ("when"), an interrogative about timing that reads as a delivery-chasing question. 30 optimizer steps are enough to teach rules that agree with the prior, but not enough to overwrite one that contradicts it.

Training

max_length=1024 is the lab's hardware-tier constant, not a tuned value; the measured token-length p95 was 98 and the longest example was 101, so nothing was truncated.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "unsloth/Qwen3.5-4B"
ADAPTER = "minhlt12/qwen3.5-4b-lora-vi-ticket-triage"

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="float16", device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

SYSTEM = "Phân loại ticket sau."
ticket = "Shop ơi, mình đặt nồi chiên không dầu mã đơn DH249548. Thiếu phụ kiện. Khi nào tiện."

msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": ticket}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
                     max_new_tokens=64, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
# -> {"intent": "san_pham_loi", "urgency": "trung_binh", ...}
#    note: gold urgency here is "thap" — this is the known failure mode above

Greedy decoding (do_sample=False) is what the reported numbers use.

Limitations

• —Fails its regression gate (see the warning at the top). Do not deploy.
• —Trained on synthetic tickets from a fixed template family. Real support text is messier, and these numbers will not transfer unchanged.
• —Vietnamese only. The label vocabulary is closed: intent ∈ {doi_tra, van_chuyen, hoan_tien, san_pham_loi, hoi_thong_tin}, urgency ∈ {cao, trung_binh, thap}, sentiment ∈ {tieu_cuc, trung_tinh, tich_cuc}.
• —30 training steps on 225 examples. This is a lab-scale run, not a converged one.
• —The eval set is small (50 target / 15 regression); treat differences under a few points as noise.

Framework versions

• —PEFT 0.20.0
• —Base model: unsloth/Qwen3.5-4B