VikramPal/Qwen3-Omni-30B-A3B-Thinker-QLoRA

Qwen3-Omni-30B-A3B — SLURP intent QLoRA adapter

A QLoRA adapter for the Thinker of Qwen/Qwen3-Omni-30B-A3B-Instruct, trained on SLURP spoken-intent classification. The adapter directory is 112,207,855 B (107 MiB), of which 100,767,320 B (96.1 MiB) is adapter_model.safetensors and roughly 11.4 MB is the copied tokenizer.json.

It is the smallest artifact of this campaign, and the only one that cannot be recomputed from anything else here: the merged bf16 checkpoint is rebuilt from this adapter plus the base weights, and the two quantized checkpoints from that merge plus the saved DynQuant bit maps.

Merged into the base Thinker and evaluated on 500 held-out SLURP test items, it scores 86.80% against the unmodified base checkpoint's 79.40%, a paired gain of +7.40 points (McNemar, p = 7.51e-07).

Which repo do you want? If you just want to run the model, take the **4-bit** repo (14.77 GiB, 86.20%, statistically tied with bf16) or the **bf16** repo (59.08 GiB, 86.80%, the reference). Take this adapter only if you want to rebuild those yourself, inspect what the fine-tune moved, or serve it unmerged over an NF4 base. The adapter is the smallest download and the largest memory requirement — see Requirements.

Scope: Thinker only, and there is no speech output

Read this before you plan anything around the model.

Only the Thinker was trained. The Talker and code2wav stacks — 3,540,613,057 parameters — were nulled out before training and are excluded from every artifact in this campaign. The adapter targets modules that exist only inside the Thinker, and the merged and quantized checkpoints published alongside it contain no Talker at all: their config.json has exactly three sub-configs, audio_config, text_config and vision_config.

The consequence is simple and absolute. The model accepts audio, images, video and text as input and emits text only. There is no speech synthesis. This is not a speech-to-speech model, and nothing here should be described as one.

Which class loads what

The merged and quantized checkpoints published here carry architectures: ["Qwen3OmniMoeThinkerForConditionalGeneration"] (model_type: qwen3_omni_moe_thinker) and are loaded with that class directly. The base repo is a different case: its experts are stored per-expert and unfused, and only Qwen3OmniMoeForConditionalGeneration carries the conversion that fuses them into the 96 batched banks — so to attach this adapter you load the full class and take .thinker, as in the snippet below.

AutoModelForCausalLM does not claim qwen3_omni_moe and will not work. AutoModel is not a substitute either: qwen3_omni_moe_thinker is absent from MODEL_MAPPING_NAMES, so AutoModel.from_pretrained raises ValueError: Unrecognized configuration class Qwen3OmniMoeThinkerConfig; the auto class this model_type is registered under is AutoModelForImageTextToText. On the base repo AutoModel returns the whole Omni model, Talker included, which is not what this adapter was fitted against. All three classes (Qwen3OmniMoeForConditionalGeneration, Qwen3OmniMoeThinkerForConditionalGeneration, Qwen3OmniMoeProcessor) ship natively in transformers 5.15.0 — no trust_remote_code anywhere.

Parameter counts, for orientation: the whole Omni checkpoint is 35,259,818,545 parameters, of which the Thinker is 31,719,205,488 and the Talker plus code2wav are 3,540,613,057. Within the Thinker, the 96 batched MoE expert banks are 28,991,029,248 parameters — 91.399% of it. tie_word_embeddings is false, so embed_tokens and lm_head are two separate 152064x2048 tensors.

Requirements

The adapter is 107 MiB. Everything expensive is the base model behind it.

Versions this was produced and verified under: transformers >= 5.0 (5.15.0 measured), torch 2.11+cu128, peft 0.20.0, accelerate (needed by device_map="auto"), datasets plus an audio backend for the audio path, bitsandbytes for the NF4 route, and pip install dynquant (0.4.0) for the two quantized repositories.

A defect in the shipped adapter_config.json

The adapter_config.json written by the training run had `base_model_name_or_path` set to the empty string `""` and `task_type` set to `null`. Both are recorded here rather than quietly fixed, because the second one is still present in the file you will download.

• —base_model_name_or_path has been patched to Qwen/Qwen3-Omni-30B-A3B-Instruct before upload. Left empty it would have broken any loader that resolves the base model from the adapter config.
• —task_type is still `null`, exactly as written. peft therefore cannot infer a task wrapper for this adapter. Construct the base model yourself with the full Omni class, take .thinker, and pass that object to PeftModel.from_pretrained, as in the snippet below. If your peft version complains about the missing task type on an AutoPeftModel* path, that is the reason; the explicit-base route avoids it entirely. The adapter was written by peft 0.20.0, and its auto_mapping.base_model_class records Qwen3OmniMoeThinkerForConditionalGeneration — the Thinker, not the whole Omni model.

Loading

Ignore the "Use this model" snippet the Hub renders above this card. library_name: peft makes it emit AutoModelForCausalLM.from_pretrained(...), which does not claim qwen3_omni_moe and cannot load this base model. Use the code below.

import torch
from transformers import AutoProcessor, Qwen3OmniMoeForConditionalGeneration
from peft import PeftModel

BASE = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
ADAPTER = "VikramPal/Qwen3-Omni-30B-A3B-Thinker-QLoRA"

processor = AutoProcessor.from_pretrained(BASE)   # native class, no trust_remote_code

# Load the FULL Omni class and take `.thinker`. Do NOT load
# Qwen3OmniMoeThinkerForConditionalGeneration against this repo: the base checkpoint
# stores the MoE experts per-expert and unfused (`experts.{e}.{gate,up,down}_proj`,
# 128 experts x 48 layers), and only the full class's conversion fuses gate with up and
# stacks them into the 96 batched 3-D banks. Asking the Thinker class for those keys
# matches 0 of 1,407 as-is and leaves all 96 banks -- 91.4% of the parameters -- missing
# and randomly initialised. `from_pretrained` reports that as a printed missing-key
# table, not an exception, so the model loads, generates, and is garbage.
# On transformers < 5 the dtype argument is spelled `torch_dtype` rather than `dtype`.
whole = Qwen3OmniMoeForConditionalGeneration.from_pretrained(
    BASE,
    dtype=torch.bfloat16,
    device_map="auto",      # ~70 GiB peak across your GPUs -- see Requirements
)
base = whole.thinker
whole.talker = None         # never trained, never shipped, never runs
whole.code2wav = None
del whole

# task_type is null in adapter_config.json, so pass the constructed base model
# explicitly rather than going through an AutoPeftModel* helper. Attach to `.thinker`
# and never to the full model: the target names q_proj/k_proj/v_proj/o_proj also match
# the Talker's projections, which this adapter was not fitted against.
model = PeftModel.from_pretrained(base, ADAPTER, is_trainable=False)
model.eval()

# The three Thinker-only repos published alongside this adapter (bf16 and the two packed
# checkpoints) ARE loaded with Qwen3OmniMoeThinkerForConditionalGeneration directly --
# their weights are already written in the Thinker's fused layout. Only the *base* repo
# needs the full class.

Inference: the prompt this adapter was trained under

The fine-tune taught one prompt and one output format. The model answers with an index into the 60-intent menu — an integer such as 37, not an intent name and not a sentence. Prompt it any other way and the 86.80% does not apply.

import numpy as np
import soundfile as sf

# `intents`: SLURP's 60 `scenario_action` labels, sorted(), numbered from 0.
# sha256 of "\n".join(intents) must be
#   d04b663b407e9f5b5be80c9d11160c391c7b68f516c9da957aaca026138fc86d
# Built by dynquant.eval.slurp.official_taxonomy() from SLURP's own annotation.
prompt = (
    "Listen to the spoken command and classify it into one of these intents.\n"
    "Answer with the number only.\n"
    + "\n".join(f"{i}. {name}" for i, name in enumerate(intents))
)

# The four exemplars are TEXT transcripts, not audio: "<transcript> -> <index>".
prompt += "\n\nExamples, as text:\n" + "\n".join(f"{t} -> {i}" for t, i in shots)

audio, rate = sf.read("command.wav", dtype="float32")
assert rate == 16000        # a wrong rate does not raise; it reads the clip at the
if audio.ndim > 1:          # wrong speed and scores like a weak model
    audio = audio.mean(axis=1)

conversation = [{"role": "user", "content": [
    {"type": "text",  "text": prompt + "\n\nNow the spoken command:"},
    {"type": "audio", "audio": audio},   # a bare array; the {"array", "sampling_rate"} dict raises
    {"type": "text",  "text": "Intent:"},
]}]
inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    sampling_rate=16000,     # passed beside the block, not inside it
).to(model.device)

out = model.generate(**inputs, max_new_tokens=8, do_sample=False, num_beams=1,
                     repetition_penalty=1.0, length_penalty=1.0, no_repeat_ngram_size=0)
text = processor.batch_decode(
    out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True
)[0]
print(text)                  # e.g. "37"  ->  intents[37]

Scoring takes a leading integer in [0, 60) if there is one, else the first in-range integer anywhere in the generation; decoding stops at the first newline. intents and the four shots are produced by dynquant.eval.slurp.load_slurp and official_taxonomy — see Evaluation.

do_sample=False alone is not enough on transformers 5.x: unset generation fields are filled from the checkpoint's own generation_config, so a shipped repetition_penalty survives and moves the score silently. The harness pins the neutral fields explicitly, and so does the snippet above.

Rebuilding the bf16 checkpoint

To reproduce the published bf16 checkpoint, merge and save. The merge moves 384 weights and leaves all 96 expert banks unchanged; the resulting shard files are 63,440,876,184 B and the directory is 63,454,086,373 B (59.10 GiB on disk; the weight tensors themselves are 63,438,410,976 B, 59.08 GiB).

# Rebuild `base` and `model` above with device_map="cpu" before running this.
# transformers v5's save_pretrained calls revert_weight_conversion, which un-fuses with
# torch.chunk(...).contiguous() on whatever device holds the weights -- this campaign
# OOM'd at 93.60 of 94.97 GiB doing it on GPU. On CPU the whole stage costs 48.1 s and
# about 70 GiB of system RAM. The merge itself needs no GPU; only the write does.
merged = model.merge_and_unload(safe_merge=True)   # safe_merge: peft rejects a NaN merge
merged.save_pretrained("qwen3-omni-thinker-slurp-bf16", safe_serialization=True)
processor.save_pretrained("qwen3-omni-thinker-slurp-bf16")

# Read the config back rather than trusting it: downstream loads resolve the class by
# name off config.architectures, and a config naming the whole Omni class over
# Thinker-only weights fails with a missing-key table, not an exception.
import json
assert json.load(open("qwen3-omni-thinker-slurp-bf16/config.json"))["architectures"] == [
    "Qwen3OmniMoeThinkerForConditionalGeneration"
]

The adapter was trained on top of an NF4-quantized base (QLoRA) and merged into bf16, which is standard for QLoRA and means the merged checkpoint is not bit-for-bit the model that produced the training-time signal. If you want to serve the adapter unmerged, an NF4 base is the configuration it was actually fitted in.

Training

QLoRA over an NF4 base with peft 0.20.0: r = 16, lora_alpha = 32, lora_dropout = 0.05, bias = none. target_modules is q_proj, k_proj, v_proj, o_proj, out_proj, fc1, fc2.

Run with torchrun at world size 2, effective batch 16, for 500 optimizer steps in 4188.1 s wall time. Final train_loss 4.031702354431152; the loss curve runs 5.479 to 3.080. Training used 8,000 of the 50,628 SLURP train recordings (with the four few-shot exemplars excluded from that pool), which between them carried 59 distinct intents out of the 60-class menu.

Hardware and stack: vast.ai, 2x RTX PRO 6000 Blackwell Max-Q (94.97 GiB each, sm_120), torch 2.11+cu128, transformers 5.15.0, dynquant-core 0.4.0.

The expert banks were not adapted

This matters more than the hyperparameters. LoRA does not reach batched 3-D expert banks. The 96 MoE banks hold 28,991,029,248 parameters — 91.399% of the Thinker — and not one of them was adapted. Under LoRA every base weight is frozen; only rank-16 adapters on 384 projections took a gradient, and the 91.4% of parameters in the expert banks carried no adapter at all. The merge record confirms it from the other side: 384 weights moved, 96 banks unchanged.

Task and data

SLURP intent classification. The label is the joint `scenario_action` pair, giving 60 classes. Audio comes from the Hub dataset `marcel-gohsen/slurp`; labels are taken from SLURP's own annotation at https://raw.githubusercontent.com/pswietojanski/slurp/master/dataset/slurp/{split}.jsonl rather than from the mirror.

The two sources are deliberate. The mirrors' own intent column is corrupted: on 1,548 of 72,396 recordings it disagrees with the scenario/action pair, having dropped the scenario, so nine different *_query intents all collapse to query. That yields 91/71/77 classes per split against the real 60. The scenario and action columns are clean, so the joint pair is the label.

Evaluation

The protocol is identical for every arm: the first 500 items of a `random.Random(0)` shuffle of the SLURP test split, 4 shots drawn from train, greedy decoding, max_new_tokens=8, stop at the first newline, add_special_tokens=False, max_prompt_tokens=4096, batch 8, MoE experts dispatch pinned to eager, the 60-intent menu (intents_sha d04b663b407e9f5b5be80c9d11160c391c7b68f516c9da957aaca026138fc86d). Chance is 1.667%. Per-item hits are stored, so every comparison is an exact McNemar test on paired outcomes.

The seed does more than name a draw, so the steps are written out. Both splits are shuffled whole with random.Random(0) at load — Hub splits arrive grouped by speaker and scenario, so any raw prefix samples one corner of the label space — and the 500 scored items are the first 500 of the shuffled test order, not the first 500 rows the mirror serves. The shots are sorted(random.Random(0).sample(range(50628), 4)) over the shuffled train pool; their gold indices are 27, 15, 34, 22, they are rendered as text "{transcript} -> {index}" under the line Examples, as text:, and those four rows are excluded from the 8,000 training rows. The menu is sorted(set(f"{scenario}_{action}")) read across train+devel+test together (no single split carries all 60), labelled from SLURP's own annotation and never from the mirror's intent column. All of it is in dynquant.eval.slurp; the exact command is

pip install "dynquant==0.4.0" "transformers>=5.15,<6" "datasets>=4"
dynquant eval qwen3-omni-thinker-slurp-bf16 --task slurp \
  --model-class Qwen3OmniMoeThinkerForConditionalGeneration \
  --limit 500 --shots 4 --shot-seed 0 --experts-impl eager \
  --out slurp-sft.json

95% confidence intervals on the paired differences: base to sft [+4.51, +10.29]; sft to dq4 [−2.64, +1.44]; sft to dq3 [−66.17, −57.43]. The omni-dq3 arm returned 3 unparseable generations; every other arm in this table returned 0.

The row that belongs to this adapter is omni-sft. The two quantized rows are included so the four repositories can be read together, and each is documented on its own card. In short: at 4.00x fewer bytes the 4-bit arm does not separate from the bf16 ceiling — the honest form of that claim is the interval, which excludes damage worse than 2.64 points but does not establish that damage is zero. The 3-bit arm is a measured collapse published as one. DynQuant's role floors alone cost 3.418 average bits on this architecture, so a 3.00-bit target sits below the floor budget: soft floors bind, the allocator downgrades by lowest ROI, 42.9% of parameters land at 2 bits and lm_head is cut from 8 bits to 3. That arm measures floor-override damage; it is not the 4-bit experiment at a lower budget.

Limitations

The expert banks were never adapted. 91.399% of the Thinker's parameters — the 96 batched MoE banks, 28,991,029,248 of 31,719,205,488 — were frozen and are unchanged by this adapter. Whatever the fine-tune achieved, it achieved on the parameters outside those banks. Anyone expecting a LoRA sweep over "all linear layers" to have touched an MoE of this shape should check the target list against the module census first.

500 evaluation items bound the resolution of every claim here. The +7.40 point gain is real and separated, but its interval is [+4.51, +10.29]; the point estimate is not precise. Nothing in this campaign supports finer distinctions than the intervals allow.

One task, one language. SLURP intent classification in English, 60 classes, and only 59 of them appeared in the 8,000 training recordings. There is no evidence here about any other task, any other label set, or any other language.

SLURP is one collection with one recording protocol. Accuracy on it does not transfer unexamined to other microphones and acoustic conditions, telephony audio, unfamiliar accents, or spontaneous speech outside this collection.

The base-to-SFT comparison differs in two things. The base arm ran the whole Omni checkpoint while every arm after the merge runs the Thinker alone, and only the post-merge arms carry the merged adapter. It remains a fair comparison — the Talker is strictly downstream of the text the Thinker emits, and SLURP scores that text — but it is not a single-variable contrast. The arms carrying the quantization claims (sft against dq4 against dq3) are all Thinker-only and do differ in exactly one thing.

Text output only. Stated again because it constrains what the adapter can be used for: no speech is produced by anything in this campaign.

No competitive baselines. GPTQ, AWQ and RTN were out of scope for this phase, so there is no matched-bytes comparison against another quantizer anywhere in these four repositories.

Related repositories

The two quantized checkpoints require the dynquant package to load correctly, and that requirement is not enforced by an exception: without it, transformers logs a warning, sets pre_quantized to false, and returns a randomly initialised model that generates fluent nonsense. Install it with pip install dynquant (0.4.0) and call `dynquant.register_hf_quantizer()` before loading either one — installing alone is not enough, because registration is an explicit call and not an import side effect. Their cards say so at the top. This adapter is unaffected, since it carries no quantization config of its own. DynQuant source: <https://github.com/kambojvikram/dynquant>.

License

The base model Qwen/Qwen3-Omni-30B-A3B-Instruct is published under license: other with license_name: apache-2.0, and this adapter mirrors that. Consult the base model's own repository for the governing terms; nothing here grants rights beyond them.

SLURP is distributed under CC BY 4.0. Its terms apply to any redistribution of the data, or of derivatives that embed it.

Citation

SLURP:

@inproceedings{bastianelli-etal-2020-slurp,
  title     = {{SLURP}: A Spoken Language Understanding Resource Package},
  author    = {Bastianelli, Emanuele and Vanzo, Andrea and Swietojanski, Pawel and Rieser, Verena},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2020}
}

The quantization tooling used by the sibling repositories is DynQuant (<https://github.com/kambojvikram/dynquant>), version 0.4.0.