SASVAAI/Qwen3.8-27b-Clinc-OOS-Lora

Qwen3.8-27B CLINC-OOS Intent Classifier (LoRA)

Classifies a short English utterance into exactly one of 150 intent labels, or oos when it matches none. For intent routing in assistants and support bots whose taxonomy matches CLINC150.

This is a LoRA adapter for Qwen/Qwen3.8-27B, trained with bf16 LoRA (no quantisation) via TRL SFT.

Model details

Trainable parameters: 108,789,760 across 400 modules — 0.396% of the base. The adapter file is 435,280,528 bytes (800 tensors: lora_A + lora_B per module). All 800 sit under model.language_model; the base model's vision tower is loaded but untouched.

Qwen 3.8 is a hybrid-attention decoder — of its 64 text layers, 48 are Gated DeltaNet and 16 are standard GQA — which is why the module list spans three naming schemes. Targeting only q/k/v/o would silently leave 48 layers unadapted.

Intended use

Direct use. Map one short English utterance to one of 150 intent labels or oos. The model was trained on a specific prompt shape and that shape is part of the contract:

• —System prompt (verbatim): "Classify the user's utterance into exactly one of 150 intent labels (snake_case) or `oos` if it does not match any supported intent. Output only the intent label on a single line, with no explanation. If you are not confident the utterance matches a known intent, output `oos`."
• —User turn: the raw utterance, nothing else.
• —Applied through the tokenizer's chat template (chat_template.jinja, shipped in this repo). Do not concatenate strings by hand.
• —The label is the first line of the generation; discard anything after it.

Out of scope.

• —Any taxonomy other than CLINC150's 150 intents. The label set is fixed by fine-tuning; the model cannot classify into intents it never saw.
• —oos is not a confidence or abstention signal — it is a 151st trained class. Do not read it as "the model is unsure".
• —Not a safety filter. oos means "outside the supported intent set", not "unsafe".
• —Long, multi-intent, or non-English inputs. Training utterances are short single-intent English sentences; behaviour elsewhere is unmeasured.
• —Not a general-purpose assistant. It emits a bare label, never prose, and will degrade on open-ended chat.

How to get started

from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoTokenizer

BASE = "Qwen/Qwen3.8-27B"
ADAPTER = "SASVAAI/Qwen3.8-27b-Clinc-OOS-Lora"

# NOTE: AutoModelForImageTextToText, not AutoModelForCausalLM. Qwen 3.8 is
# multimodal; its CausalLM mapping resolves to the text-only Qwen3_5ForCausalLM,
# which receives the multimodal wrapper config and raises
#   AttributeError: 'Qwen3_5Config' object has no attribute 'vocab_size'
tokenizer = AutoTokenizer.from_pretrained(ADAPTER, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    BASE, dtype="bfloat16", device_map="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

SYSTEM = (
    "Classify the user's utterance into exactly one of 150 intent labels "
    "(snake_case) or `oos` if it does not match any supported intent. Output "
    "only the intent label on a single line, with no explanation. If you are "
    "not confident the utterance matches a known intent, output `oos`."
)
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "is it possible to change to original settings"},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

out = model.generate(inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True).split("\n")[0].strip())
# -> reset_settings

The base model is ~52 GB in bfloat16 — this needs multiple 80 GB-class GPUs, or 4-bit quantisation on one.

Decoding matters. This model was evaluated with greedy decoding (do_sample=False, max_new_tokens=64). Sampling will not reproduce the reported numbers.

Training details

Data. A 13,725-utterance training split derived from the CLINC150 out-of-scope intent corpus, produced by this project's data-generation stage. Short single-sentence English utterances labelled with one of 150 intents or oos. The split is not published, so the datasets key is omitted rather than pointing at a dead link. The validation split used for the numbers below is a disjoint 1,525 utterances covering 151 distinct labels.

Method

bf16_lora is the only method this project allowed for this run (task_config.ALLOWED_TRAINING_METHODS = ['bf16_lora']). It applies LoRA to an unquantised bf16 base — the only one of the five methods that does no quantisation.

No refinement stage ran; the published weights are the SFT adapter.

Final hyperparameters

Effective batch size: 28 (1 x 4 x 7). Optimizer steps: 982.

KD parameters are omitted deliberately — this is a bf16_lora run, not bf16_lora_kd, so kd_alpha/kd_beta/kd_temperature carry inert defaults that would imply distillation that did not happen.

How these values were chosen

These hyperparameters were selected by an automated search (autocatalyst.cli.run_autoresearch): an agent proposes one change at a time, runs train → eval, and keeps or discards on f1_macro.

Read the next paragraph before quoting "best of 25".

Of 25 evaluated trials, exactly one produced a metric. The other 24 failed before scoring, for reasons unrelated to their hyperparameters (a harness output-buffer bug, a wall-clock timeout, and then 22 consecutive CUDA OOMs caused by leaked GPU processes from the earlier failures). So this configuration is not a search winner — it is the only configuration that completed. There is no second successful trial to compare it against, and no evidence that any other point in the space is worse.

In particular, the MAX_SEQ_LEN: 4096→2048 change recorded against the kept trial cannot be read as an improvement: the 4096 run (trial 1) errored rather than scoring, so 2048 has no measured competitor. Treat these values as a working baseline, not a tuned optimum.

Search space. Bounds from core/experiment_constraints.py as of this run:

Five of the fourteen knobs were never varied. EPOCHS has since been capped to 1–2: two epochs already consumes ~96% of the harness timeout, so every EPOCHS>=3 trial above was guaranteed to be killed.

The configuration that produced these weights. From results.tsv:

The other 24 trials are omitted: none of them reached scoring, and none failed for a reason connected to its hyperparameters. One hit a harness output-buffer bug, one exceeded the wall-clock timeout, and the remaining 22 died in CUDA OOM caused by GPU processes leaked by those earlier failures. They are harness defects, not negative results about the search space, and tabulating them would imply evidence that does not exist.

Outcome. No hyperparameter change is responsible for the score, because only one configuration ever scored. All 25 trials ran against the same data split, so they are comparable in principle — there is simply nothing to compare. The search was still at its first data point when it stopped, and the space is effectively unexplored.

Observed training metrics.

Loss falls from 2.5728 at step 10 to ~0.17 by step 100, then improves slowly to 0.1317 at step 980. Most of the task is learned in the first fifth of epoch 1; the second epoch buys roughly 0.02 of training loss.

Evaluation

Protocol. All 1,525 validation utterances, no sampling. Predictions generated greedily (do_sample=False, max_new_tokens=64) through the same chat template used in training. The predicted label is the first line of the generation, stripped. No constrained decoding was applied — the model was free to emit any string, which matters (see Limitations). Scored with scikit-learn accuracy_score / f1_score; f1_macro uses sklearn's default label set, which is the union of gold and predicted labels.

Baseline for comparison. Not measured. The untuned Qwen/Qwen3.8-27B was never scored on this split, so the numbers above quantify the fine-tuned model's performance but do not establish how much of it the fine-tuning is responsible for.

This is a validation split, not a held-out test set. The autoresearch loop selects against it, so expect some optimistic bias. In this run only one trial scored, so the selection pressure was minimal — but the split is still not untouched.

The eval set is the one embedded in `predictions.jsonl`, not a file you can look up. The split is a generated artifact, and the generator overwrote it (2026-08-31 19:42) after these numbers were produced — the regenerated file shares only 155 of its 1,525 rows. The input/gold pairs shipped in predictions.jsonl are therefore the authoritative record of what was scored, and the reason that file is included rather than merely offered.

Limitations and bias

The model invents labels outside the taxonomy. On 1,525 utterances it produced 156 distinct labels where only 151 exist (150 intents + oos). Five predictions were plausible but non-existent intents, one occurrence each: account, clock_in, customer_service, parental_controls, privacy.

That is 5 of 1,525 predictions — 0.33% — and it accounts for essentially the entire macro/micro gap, because macro-F1 averages over every label appearing in gold or predictions and each invented label scores 0:

Exactly: 0.987538 x 151/156 = 0.955886.

Mitigation: constrain decoding to the known label set, or map anything unrecognised to oos. This should recover most of the 0.032 gap.

Per-class performance is uneven. F1-macro is the primary metric precisely because it exposes this. The twelve weakest classes:

Support is small — most classes have 8–14 validation examples, so one error moves a class F1 by ~0.05. Read individual per-class figures as indicative.

`oos` is conservative. Precision 1.000, recall 0.875 over 32 examples: when it says oos it is right, but it misses 4 of 32 genuinely out-of-scope utterances and assigns them a real intent instead. Where a confidently wrong intent is costlier than an unnecessary fallback, add the label-set guard above.

Domain narrowness. Short single-sentence English assistant utterances only. Expect degradation on longer inputs, compound requests, other domains, and other languages — none of which were measured.

Inherits all biases and limitations of the base model. This adapter changes 0.396% of the parameters and was not evaluated for social bias, safety, or fairness across demographic groups.

Merged-weights equivalence

A merged build of these weights (base + adapter folded into one standalone model, W + (alpha/r) * B @ A in bf16) was evaluated on the identical split:

Exactly 1 of 1,525 predictions differs (0.07%) — one borderline utterance where bf16 rounding flipped a near-tie. Merging is exact here because the adapter was trained on an unquantised bf16 base; a QLoRA adapter would not reproduce this.

So a merged distribution of this model is behaviourally equivalent. Apache-2.0 Sec.2 permits distributing Derivative Works, so publishing one is allowed — note that it would also redistribute the base model's vision tower, which this text-only classifier never uses.

Environmental impact

Covers this run only; it excludes the 24 failed trials and the wider search.

Framework versions

• —PEFT 0.18.1
• —TRL: 1.0.0
• —Transformers: 5.7.0.dev0
• —Pytorch: 2.5.1+cu121
• —Datasets: 4.8.4
• —Tokenizers: 0.22.2

transformers is a git-main build: Qwen 3.8's qwen3_5 architecture is not in the stable PyPI release.

Citation

@misc{qwen38_clinc_oos_lora_2026,
  title  = {Qwen3.8-27B CLINC-OOS Intent Classifier (LoRA)},
  author = {Banerjee, Aaron and Anbuselvan, Pooja and Jodhpurkar, Om},
  year   = {2026},
  url    = {https://huggingface.co/SASVAAI/Qwen3.8-27b-Clinc-OOS-Lora}
}