Modelpublic

AygerimTan/eva-qwen35-4b-support-merged

sourceHugging Faceupdated 5d agoView on Hugging Face

0likes377downloads

Model Card

Eva Customer Support Manager — merged model

Qwen/Qwen3.5-4B adapted for customer-support triage in a women's health app (cycle tracking, pregnancy, teen mode) operating in Kazakhstan in Russian and Kazakh. A QLoRA adapter was trained separately and merged into the unquantized half-precision base.

Given one customer message, the model emits only a JSON object classifying it on four axes. It does not write prose and is not meant to: in the system it was built for, the JSON selects a reply from an approved template library.

Intended use

Triage and routing. Downstream of this model sits a deterministic safety layer that can only reduce the system's autonomy, never raise it. Do not deploy the model without that layer. It is not a medical device, not a diagnostic tool, and must never answer a medical question on its own.

Output format

json

{
  "category": "auth_otp",
  "priority": "high",
  "sentiment": "negative",
  "action": "answer_from_kb"
}

Field	Allowed values
`category`	`auth_otp` · `auth_account` · `age_guardian` · `geo_availability` · `cycle_predictions` · `pregnancy` · `health_diary` · `app_technical` · `store_order` · `store_payment` · `store_promo` · `store_product` · `medical_concern` · `other`
`priority`	`low` · `medium` · `high` · `critical`
`sentiment`	`positive` · `neutral` · `negative`
`action`	`answer_from_kb` · `explain_expected_behaviour` · `request_details` · `create_bug_ticket` · `escalate_human` · `escalate_medical` · `escalate_legal_privacy`

The label space was frozen after measuring 214 real support conversations and must not be extended: the baseline and the fine-tuned model are scored against an identical set, or the comparison below is void.

Training


Method	QLoRA — base loaded in 4-bit NF4, adapter trained, then merged into `float16`
LoRA	`r=16`, `alpha=32`, `dropout=0.05`, `target_modules="all-linear"`
Schedule	3 epochs, lr `2e-4`, batch 1 × grad-accum 8, max length 1024, seed 42
Final validation	`eval_loss` 0.560, `mean_token_accuracy` 0.900
Merge dtype	`float16` (not quantized; GGUF conversion is a separate step)
Split	train 629 · validation 207 · test 215

The test split was not used during training.

Training data

Synthetic. The corpus was generated from a catalogue of the app's real defects and a catalogue of behaviours that look like defects but are not — for example an empty "Today" screen after childbirth, or an auto-cancelled unpaid order. That second catalogue is the point: an off-the-shelf model confidently files false bug reports against intended behaviour, and no public corpus contains this knowledge.

Real customer messages were never used for training. They exist only as a held-out slice for measuring behaviour on live traffic.

Evaluation

215 held-out examples. Base model is Qwen/Qwen3.5-4B zero-shot with the same prompt.

Metric	Base	This model
Category accuracy	0.00 %	82.79 %
Action accuracy	0.00 %	69.77 %
Priority accuracy	7.44 %	59.53 %
Sentiment accuracy	7.44 %	56.28 %
Exact match, all four fields	0.00 %	35.81 %
Invalid JSON rate	86.05 %	0.00 %

Read these numbers with two caveats

1.The base model mostly failed on format, not on understanding: 185 of its 215 answers were invalid JSON. A large share of the category gain is format compliance. Reporting the category number without this line overstates what was learned.
2.The test split is synthetic. A separate slice of 282 hand-labelled real conversations exists and has not been scored yet. Until it is, nothing here is evidence about live traffic.

Class coverage in real data

Four of the fourteen categories have no real examples at all (pregnancy, health_diary) or exactly one (auth_account, age_guardian — the latter safety-critical). Per-class claims about those four are not supportable, and a macro average that hides them would be misleading.

Limitations

—Educational prototype, not a production system and not a medical device.
—Trained on synthetic data; measured on synthetic data.
—Kazakh is under-represented: 18 of 214 real conversations. The system it belongs to never grants Kazakh messages autonomous replies for this reason.
—Deidentification tooling for Kazakh does not exist, and for Russian the only benchmark was machine-translated. Do not assume input is safely scrubbed.
—Prompt injection is bounded by architecture, not by the model: the agent selects from approved templates, so the worst case is a wrong template rather than arbitrary text. Remove the template layer and that bound is gone.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AygerimTan/eva-qwen35-4b-support-merged"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    dtype="auto",
)

License

Inherits the base model's licence. Owner: confirm `Qwen/Qwen3.5-4B`'s terms and add the matching `license:` field to the front matter before treating this card as final. It is deliberately left out rather than guessed.