Modelpublic

adventists-ai/DuplexJev-B-Para-Qwen3-ASR-0.6B-Qwen3-1.7B

sourceHugging Facecc-by-nc-4.0updated 3d agoView on Hugging Face

0likes6downloads

Model Card

DuplexJev-B-Para-Qwen3-ASR-0.6B-Qwen3-1.7B

A small DuplexJev connector: the Qwen3-ASR-0.6B encoder (frozen) feeds a frozen Qwen3-1.7B, and every typed question is read as a single-token distribution over its options, with zero decode steps and many questions per forward pass. It is the edge-sized counterpart of the Qwen3-32B connectors released with the paper, trained with the same recipe.


Audio encoder (frozen)	`adventists-ai/Qwen3-ASR-0.6B-Encoder`
LLM (frozen)	`Qwen/Qwen3-1.7B`
Connector	last encoder layer → frame stacking + SwiGLU projector (B), 11.5 M trainable parameters
Audio tokens	Qwen3-ASR at 12.5 Hz, 2 frames stacked → 6.25 audio tokens/s
Total parameters	1917 M (encoder + LLM + connector)
This variant	R2 connector + 4k steps of the four-task mixture (transcription, continuation, gender, emotion) under the mixed objective: transcript distillation for content samples, answer-token cross-entropy for decision samples (the paper's MIX-KD recipe). Hears speaker gender and four-way emotion.

Quick start

bash

pip install "duplexjev[speech]>=0.2.1"

python

from duplexjev import Decider, Question

d = Decider.from_pretrained("adventists-ai/DuplexJev-B-Para-Qwen3-ASR-0.6B-Qwen3-1.7B")          # fits on one small GPU or CPU

d.decide("utterance_en.wav", [
    Question("gender", "What is the perceived gender of the speaker?", ["female", "male"], lang="en"),
    Question("emotion", "What is the speaker's emotional state?", ["neutral", "happy", "angry", "sad"], lang="en"),
], lang="en")
d.decide("utterance_zh.wav", [                            # Chinese speech: Chinese question and options
    Question("gender", "说话人的性别是？", ["男性", "女性"], lang="zh"),
    Question("emotion", "说话人当时的情绪状态是？", ["中性", "高兴", "生气", "伤心"], lang="zh"),
], lang="zh")

Results

Evaluation (%)	paper protocol	`duplexjev` 0.2.1 defaults
qa100 (spoken QA)	59	51
qa100, Qwen3-1.7B reading the transcript	66	–
ZJU-ML (spoken QA, real + TTS)	39	–
Gender, 800 real utterances	84.8	88.5
Emotion, 4-way, 800 utterances	89.1	88.4

Paper protocol: the evaluation prompts of the paper (single option order, one H200). duplexjev defaults: the package's own prompts with lang matched to the clip. Spoken QA with a small LLM is sensitive to the prompt wording, and it is bounded by the LLM itself (see the transcript-reading row); use these connectors for short decisions and speaker cues rather than open knowledge questions. For comparison, the Qwen3-32B connectors reach 90 (qa100), 89.9 (gender) and 90.0 (emotion); see `adventists-ai`.

Latency of one decision event (10 questions, one 4.5 s clip, one forward pass, duplexjev 0.2.1, plain PyTorch): 44 ms on one H200 (bf16); 2142 ms on 8 CPU threads (Xeon Platinum 8558, fp32, not quantized). On-device NPUs and quantization have not been tested yet.

Training

Frozen encoder and LLM; only the connector is trained, with the paper's recipe: R1 and R2 transcript distillation on two disjoint 0.5 M-utterance packs of the Ultravox v0.6 mixture (32k steps each, global batch 16), then 4k steps of MIX-KD on content + gender (AISHELL-1, LibriSpeech) + emotion (ESD, CREMA-D). Code and scripts: https://github.com/adventists-ai/duplexjev.

Limitations

—Evaluated on read or acted speech and small test sets (100–800 items); not on streaming input.
—Small LLMs answer knowledge questions poorly even from the transcript.
—Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.
—The connector only works with the encoder and LLM listed above.

License

CC BY-NC 4.0, research use only. Trained on the ESD emotional speech corpus, which is licensed for research only, and on CREMA-D. Do not use it for commercial purposes. The frozen base models keep their own licences.

Citation

bibtex

@inproceedings{jin2027duplexjev,
  title     = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
  author    = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
  booktitle = {Submitted to IEEE ICASSP},
  year      = {2027}
}

Built by Adventists.ai. Claude (Anthropic) assisted with code.