DuplexJev-32B-Para (v1.5)

📄 Paper: arXiv:2610.02638 · Code: github.com/adventists-ai/duplexjev · Release notes: hearing the whole clip (v1.5) · Japanese as a third language · weak languages · Research notes: hearing stress · keeping a speech-decision LLM able to talk

🗣️ Now with a voice: DuplexJev-32B-Para is the ears of Sophon-Realtime v1, our open full-duplex voice model that hears your tone and speaks with one. Released: Tag and Flash, one GPU each.

The DuplexJev model that hears the most beyond the words. On top of everything DuplexJev-32B-Turn and DuplexJev-32B-Stress do (spoken QA, gender, emotion, turn-taking, stress and pauses), it also hears laughter, breathing, coughs and sighs, notices how someone is talking (whispering, shouting, very fast or slow), and can tell, roughly as a person would, whether two clips are the same speaker. Mandarin and English. Since v1.1 it also holds a normal conversation, multi-turn included. v1.2 adds accent judgments (English accents and Mandarin dialect regions) and much sharper end-of-turn detection right after the user pauses: on LiveKit's eot-bench it now beats Smart Turn v3.2 in English and Chinese. v1.3 hears stress much better: on the public StressTest benchmark it goes from 72 to 85, on StressPresso's meaning questions from 69 to 89, and on choosing the meaning a stressed word implies (DEBATE, held-out) from 85 to 93, with the other decisions about the same. v1.4 understands far more languages: Japanese error goes from 33 % to 12 % (FLEURS) and from 38 % to 11 % on conversational Japanese, Korean and Vietnamese error falls to a quarter to a third of v1.3, Thai, Arabic and Russian error by 30–40 %, and Mandarin conversational paralinguistics (laughter, breath, drawl, stutter …) are much sharper. v1.5 hears the whole clip: earlier versions (like every Qwen3-ASR-based checkpoint we trained) dropped the last ~4 % of every 30 s audio chunk, because the connector assumed 6.25 audio tokens per second while the encoder emits 6.5. v1.5 is retrained on the true length, which sharpens the decisions that depend on the end of an utterance: end-of-turn (Smart Turn test set 93.3 → 94.4, Chinese 91.4 → 94.6; LiveKit eot-bench English false cut-offs at 300 ms 20.4 → 17.7 %), stress (StressTest 81.7 → 85.3), and emotion in natural speech (LISTEN emotion 32.4 → 33.2, LISTEN non-verbal 48.8 → 51.6), with spoken QA up a point as well.

v1.5 (2026-11-03) replaces v1.4 (2026-10-22): audio is read at its true length ("audio_full_length": true in config.json). Use duplexjev-vllm>=0.4 / duplexjev>=0.7, which read this flag; older versions still load the model but cut the last ~4 % of each 30 s chunk, as before. Earlier versions stay available as revisions v1.4, v1.3, v1.2, v1.1 and v1.0 (--revision v1.4 in vLLM, revision="v1.4" in from_pretrained).

DuplexJev reads typed, closed-set decisions from speech as a single-token distribution of the LLM: zero decode steps, many questions about the same clip in parallel. This repository is the complete model (Qwen3-ASR-0.6B encoder

  • connector + the language model of Qwen3-VL-32B, 33.0 B parameters, bf16; one 80 GB GPU).

Which 32B to pick. This one, for everything above.

What changed

Starting from DuplexJev-32B-Turn we trained the connector and a new rank-16 LoRA on the LLM (merged here) for 4,000 steps. The mix is the 32B-Turn mix plus:

added data what it teaches
stress / pause detection (as in 32B-Stress: DEBATE, Stress-17K, TinyStress-15K, Expresso, neutral negatives) which word is stressed, or none; where the pause is
speaker pairs (VoxCeleb1 dev, AISHELL-1, LibriSpeech, two-speaker mixtures) same speaker or not; is this person in the clip
non-verbal sounds (VocalSound, NonVerbalSpeech-38K, NonverbalTTS, Expresso laughing) is there laughter / breathing / a cough / a sigh / sniffing / throat clearing / a sneeze / crying …; negatives are cut from the same recordings away from the event
speaking style (EARS, Expresso whisper) whispering, loud, fast, slow, high or low pitch, normal
spoken multiple-choice replay (ARC, OpenBookQA and CommonsenseQA train splits read by TTS) keeps "answer the content" from being crowded out by "listen to the voice"
v1.1: conversation replay (spoken and typed user turns, up to 6 earlier turns as history; target = the original LLM's own reply) and in-house chat items normal, full-length replies; trained with per-row loss averaging so long replies do not swamp the one-token decisions
v1.2: accents (VCTK, English Dialects, EdAcc, Common Voice accent labels, KeSpeech; held-out speakers kept for testing) which accent: 19 English accents and 9 Mandarin dialect regions; native speaker or not; standard Mandarin or not; which English region
v1.2: end of turn right after a pause (otoSpeech-141h cut 0.2 s into each pause, with DualTurn labels; Smart Turn v3.2 training set) finished vs. still talking / hesitating when the user has just gone quiet, the way a live voice agent has to decide
v1.3: the stress data re-balanced (more "which meaning does this stress imply" items, weight raised) and the gender data weighted ×2; no new corpora choosing the meaning a stressed word implies; keeping gender where it was
v1.4: weak-language speech (ja, ko, vi, th, ar, ru: Emilia, Emilia-YODAS, YODAS2, Zeroth-Korean, VIVOS, FLEURS train, Common Voice; 162 k, labels re-checked with Qwen3-ASR-0.6B) + continuation (36 k) transcribe and understand these languages
v1.4: Japanese speech (Emilia JA, Emilia-YODAS JA, Common Voice ja and other Japanese speech data; 358 k, re-checked) + Japanese continuation (20 k) Japanese close to Chinese and English
v1.4: Mandarin conversational paralinguistics (licensed MagicData corpus, training speakers only; 42 k questions) laughter, laughing while talking, breath, lip smack, sniff, cough, stressed word, drawl, which character is drawn out, stutter, sudden noise, unnatural pause, sentence emotion
v1.4: more emotion classes (surprise, fear, anxiety, excitement, encouragement, apologetic, playful) from CSEMOTIONS, EmotionTalk, DailyTalk, JL-Corpus, EMNS, Emozionalmente, SUBESCO emotions beyond the basic four
v1.5: the whole v1.4 mix again, now with audio tokens at the encoder's true length (6.5 per second); transcription items with native-language instructions and leading / trailing silence (FLEURS dev split, the v1.4 weak-language speech); a light replay of all emotion data the end of every clip is heard; always transcribe when asked to; emotion kept sharp

v1.4 was trained from v1.3 in four short stages (2,000 + 2,000 + 1,500 + 1,000 steps; connector + a new rank-16 LoRA each time, merged here), replaying the full v1.3 mix in every stage; the last two stages raise the stress, non-verbal, emotion and speaking-style replay ×1.5 each.

v1.5 was trained from v1.4 in two stages (2,000 + 1,500 steps; connector + a new rank-16 LoRA each time, merged here) with the audio-token count computed from the encoder's real output length.

v1.5 vs v1.4

v1.4 v1.5
qa100 / ZJU-ML (vLLM, client wording) 97 / 90 98 / 91
gender / emotion (vLLM) 92.1 / 89.2 92.6 / 90.0
Easy-Turn* (vLLM) 94.1 95.4
Smart Turn v3.2 test set, all / Chinese (threshold 0.5) 93.3 / 91.4 94.4 / 94.6
eot-bench, English false cut-offs at 300 / 600 ms (with the assistant's previous message, fast mode) 20.4 / 8.9% 17.7 / 7.0%
eot-bench, Chinese, same setting 42.3 / 13.8% 41.0 / 13.3%
StressTest / StressPresso (which meaning the stress implies) 81.7 / 88.6 85.3 / 89.1
DEBATE: stressed word / pause position 89.8 / 79.8 90.7 / 81.4
LISTEN (EACL 2026) emotion / non-verbal, audio only 32.4 / 48.8 33.2 / 51.6
EmotionTalk / CSEMOTIONS emotion (test splits)† 62.6 / 89.8 68.5 / 94.2

Same harness for both columns (stress, LISTEN and emotion rows: single-token readout, PyTorch, % correct). † In-domain: the training splits of these corpora are in the replay data.

v1.4 vs v1.3

Languages (FLEURS test, 150 utterances per language, median word / character error, lower is better):

ja ko vi th ar ru zh en es fr de
v1.3 32.7 31.8 41.7 29.1 27.0 10.6 3.8 1.5 4.4 8.5 5.3
v1.4 11.6 7.6 14.6 18.0 19.4 6.9 3.8 0.0 3.5 7.0 5.3
encoder ceiling (Qwen3-ASR-0.6B) 7.0 2.7 7.8 5.1 16.7 7.8 2.0 1.3 3.4 6.2 5.6

Japanese, conversational (held-out Emilia JA shard) / Common Voice ja test, character error: v1.3 38.2 / 33.3 → v1.4 11.1 / 13.0 (encoder 8.3 / 10.8).

Mandarin conversational paralinguistics (licensed MagicData corpus, held-out speakers, accuracy %; yes/no unless noted):

breath lip smack laughter laughing while talking sniff cough drawl stressed word (4-way) emotion (4-way)
v1.4 91.7 85.8 89.3 87.8 87.5 71.4 79.4 85.3 70.7

(v1.3 was not trained on this corpus; the 4B-Para v1.1, also untrained on it, scores 50–66 on the yes/no rows.)

Accent, same-speaker and turn-taking (eot-bench, Smart Turn) were not re-run for v1.4; the numbers in the next sections are v1.3's.

Results (v1.3 and earlier)

32B-Turn 32B-Para v1.1 32B-Para v1.2 32B-Para v1.3
qa100 / ZJU-ML 96 / 87 96 / 88 95 / 87 96 / 87
gender / emotion 91.5 / 91.1 92.9 / 91.5 93.9 / 90.1 93.8 / 92.1
VoiceBench OBQA‡ / MMSU 85.5 / 72.1 94.5 / 75.5 94.3 / 75.0 94.1 / 75.7
Easy-Turn* 95.2 94.8 95.5 94.4
which word is stressed, "none" offered: DEBATE (zh) / StressPresso (en) 35 / 1 93 / 96 92 / 91 95 / 96
neutral speech → "no word is stressed": AISHELL-1 / LibriSpeech 92 / 50 100 / 99 99 / — 98 / 83
meaning from stress: DEBATE held-out / StressTest 35 / 52 85 / 76 85 / 72 93 / 85
StressPresso, which meaning the stress implies (English) — — 69 89
laughter / breathing / cough / sigh (AUC, NonverbalTTS test, real speech, unseen) — 0.88 / 0.81 / 0.85 / 0.80 0.87 / 0.81 / 0.84 / 0.75 ≈ v1.2†
laughter / breathing in natural Mandarin conversation (AUC, unseen third-party sample) — 0.83 / 0.65 0.80 / 0.56 ≈ v1.2†
speaking style, 7 classes (EARS held-out speakers) — 82% 79% 80%
same speaker? EER, VoxCeleb1 random / hard pairs (our harness) — 28.2% / 39.6% 31.2% / 42.3% 29.1% / 38.4%◊
accent, 4 choices: English (19 accents) / Mandarin (9 dialect regions)§ — 39.6 / 25.7 56.0 / 48.2 —
**native English speaker? / standard Mandarin?**§ — 84.8 / 46.6 91.1 / 91.3 —
LISTEN (EACL 2026), mean of the 4 audio-only conditions, full set — — 38.4 38.9

Paper protocol (single-token readout, PyTorch), % correct unless marked; AUC 0.5 = chance. Through vLLM with duplexjev-vllm in the client wording, v1.3: qa100 97, ZJU-ML 90, gender 92.1, emotion 91.8, Easy-Turn 94.5 (v1.2: 98, 90, 93.6, 89.8, 95.5; v1.1: 97, 91, 93.2, 90.0, 94.8; qa100 and ZJU-ML have 100 items each). "—": not re-run for that version.

  • † v1.3 non-verbal sounds were checked by accuracy on the same test sets (NonverbalTTS, NonVerbalSpeech-38K, VocalSound, Expresso, MagicData): within a few points of v1.2 either way; AUC not re-run.

  • ◊ Same harness re-run for v1.2: 30.7% / 38.2% (the v1.2 column is from an earlier run).

  • Accents for v1.3 were checked only on the cross-corpus set: standard Mandarin or not 88.8 (v1.2 93.1, 160 items), Mandarin dialect 4-way 61.3 (60.0); English accents stay near chance cross-corpus for both versions.

  • * Easy-Turn is in-domain (its training split is in the mix).

  • § Accent test items come from the same corpora as the training items (different speakers); chance is 25% / 50%.

  • ‡ OpenBookQA's train split is in the replay data, so the OBQA gain is partly in-domain; MMSU is unrelated to it.

  • For reference, an AudioSet AST tagger scores 0.71 / 0.75 / 0.85 / 0.76 on the same four sounds.

  • Speaker: a dedicated speaker-verification model reaches ~1% EER on VoxCeleb1; this model is meant to have a rough, human-like sense of "same voice or not", not to identify people.

Turn-taking benchmarks (v1.2 / v1.3; v1.5 in the table above)

LiveKit eot-bench (validation set, decision 0.2 s into each pause, official metrics). False cut-offs when the agent may wait at most 300 / 600 ms (lower is better):

English Chinese
LiveKit Turn Detector v1 9.9 / 4.5% 20.6 / 8.7%
DuplexJev-32B-Para v1.2, with the assistant's previous message (plugin default, full eot-bench run) 20.0 / 9.4% 37.7 / 12.2%
DuplexJev-32B-Para v1.2, with the assistant's previous message, readout="act" (fast mode) 18.4 / 7.9% 37.3 / 12.3%
DuplexJev-32B-Para v1.2, audio only (fast mode) 23.5 / 11.1% 40.0 / 12.8%
DuplexJev-32B-Para v1.3, with the assistant's previous message (fast mode) 20.3 / 8.8% 35.5 / 12.3%
DuplexJev-32B-Para v1.3, audio only (fast mode) 24.7 / 11.2% 38.8 / 12.0%
DuplexJev-32B-Turn, audio only (fast mode) 29.8 / 13.9% 44.9 / 14.5%
Smart Turn v3.2 35.2 / 14.8% 42.9 / 13.2%

eot-bench audio is not in our training data. "With the assistant's previous message" puts The assistant had just said: "..." before the audio; the Pipecat and LiveKit plugins send it (pipecat-duplexjev>=0.3, livekit-plugins-duplexjev>=0.2, gateway duplexjev>=0.6). Plugin default: the turn question, P(finished) + P(backchannel); run with every eot-bench decision point (adapter benchmarks/turn_detection/eot_bench/duplexjev_ctx_adapter.py on GitHub). readout="act" (pipecat-duplexjev>=0.4, livekit-plugins-duplexjev>=0.3): question "What should the assistant do now?", P(reply now) + P(short acknowledgement). Rows marked fast mode ask once per pause, at the 0.2 s score point.

Pipecat Smart Turn v3.2 test set (accuracy at threshold 0.5): v1.2 English 94.0% (Smart Turn v3.2: 94.7%), Chinese 91.1% (90.5%); v1.3 94.2% / 91.4%; DuplexJev-32B-Turn 65.2% / 84.6%. The Smart Turn v3.2 training set is in our training mix, so this is an in-distribution result.

Serve with vLLM

pip install "vllm[audio]>=0.29" "duplexjev-vllm>=0.4"
vllm serve adventists-ai/DuplexJev-32B-Para --max-model-len 4096

duplexjev-vllm>=0.4 reads "audio_full_length": true from config.json and feeds the whole clip; DJ_FULL_LEN=0 forces the old truncated length, DJ_FULL_LEN=1 the full length for any checkpoint.

Prompt format and client are the same as for DuplexJev-32B-Turn; see examples/vllm_client.py. Examples:

The user said: <|audio|>

Is there laughter in this audio?

Options:
A. Yes
B. No

Answer with the letter only.
用户说:<|audio|>

除了说话,这段音频里还有哪种声音?

选项:
A. 叹气
B. 咳嗽
C. 笑声
D. 都没有

请只回答选项字母。

Always offer a "none" option for stress and non-verbal questions; the model was trained with it.

Limitations

  • Accent and same-speaker scores were not re-measured for v1.4 / v1.5.
  • Thai and Arabic are still 2–3× the encoder's own error.
  • Non-verbal training audio is mostly acted (radio drama, animation, studio recordings); in natural conversation the scores are lower (see the table).
  • Breathing in natural Mandarin conversation is weak (AUC 0.56 in v1.2, 0.65 in v1.1, 0.78 in v1.0); acted / read speech is much better.
  • Accent answers describe how someone sounds, not where they are from or who they are; accents are often mixed or ambiguous. Do not use them to make decisions about people.
  • Asked about itself, it may describe abilities (seeing images, speaking) that this checkpoint does not have on its own: it takes speech and text and replies in text.
  • Speaker judgments are coarse (EER 18–40%) and must not be used to identify or verify people.
  • Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.

License

CC BY-NC 4.0, non-commercial use only.

  • Non-commercial data:
    • Stress: DEBATE (github.com/SmileHnu/DEBATE), Stress-17K and TinyStress-15K (slprl);
    • Expresso (Meta), EARS (Meta) and NonVerbalSpeech-38K, all CC BY-NC 4.0;
    • ESD (research only); SmoothConv (CC BY-NC).
    • Accents (v1.2): KeSpeech (non-commercial research use).
    • Conversation replay (v1.1): user turns from wilzzzz/paralinguistic_dialogues (no licence stated on the dataset card; used for non-commercial research), SmoothConv (CC BY-NC), AISHELL-1 (Apache-2.0), LibriSpeech (CC BY 4.0); target replies generated by the base LLM (Apache-2.0); spoken chat questions synthesised with CosyVoice2 (Apache-2.0).
  • Added in v1.4 (non-commercial or share-alike): Emilia JA/KO (CC BY-NC 4.0); VIVOS (CC BY-NC-SA 4.0); EmotionTalk (CC BY-NC-SA 4.0); DailyTalk (CC BY-SA 4.0); a licensed Mandarin conversational corpus with paralinguistic annotation (MagicData, purchased licence; audio not redistributed); other Japanese speech data used for non-commercial research.
  • Added in v1.4 (other): Emilia-YODAS JA/KO (CC BY 4.0), YODAS2 (CC BY 3.0), Zeroth-Korean (CC BY 4.0), FLEURS train split (CC BY 4.0; the test split was never used), Common Voice 17 ja/ar/ru (CC0), CSEMOTIONS (Apache-2.0), EMNS (Apache-2.0), JL-Corpus (CC0), Emozionalmente (CC BY 4.0), SUBESCO (CC BY 4.0). Labels were re-checked, and for unlabelled YODAS2 clips produced, with Qwen3-ASR-0.6B (Apache-2.0); continuation targets were written by Qwen3-32B (Apache-2.0).
  • Added in v1.5: FLEURS dev split (CC BY 4.0; the test split was never used).
  • Other data:
    • VocalSound (CC BY-SA 4.0), NonverbalTTS (audio from VoxCeleb and Expresso);
    • VoxCeleb1 (CC BY 4.0), AISHELL-1 (Apache-2.0), LibriSpeech (CC BY 4.0);
    • ARC (CC BY-SA 4.0), OpenBookQA, CommonsenseQA (MIT).
    • Accents (v1.2): VCTK (CC BY 4.0), English Dialects (CC BY-SA 4.0), EdAcc (CC BY-SA 4.0), common-accent from Common Voice (CC0 audio).
    • Turn-taking (v1.2): otoSpeech-141h (CC BY 4.0) with DualTurn labels; Smart Turn v3.2 training set (pipecat-ai, CC BY 4.0).
  • Base models keep their licences: Qwen3-VL-32B-Instruct and Qwen3-ASR-0.6B are Apache-2.0 (Alibaba Cloud).
  • The LLM weights are the language model of Qwen3-VL-32B-Instruct with LoRAs merged in; the vision encoder is not included.

Citation

@misc{jin2026duplexjev,
  title         = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
  author        = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
  year          = {2026},
  eprint        = {2610.02638},
  archivePrefix = {arXiv},
  note          = {Submitted to IEEE ICASSP 2027}
}

Built by Adventists.ai. Claude (Anthropic) assisted with code.

Downloads last month
346
Safetensors
Model size
33B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for adventists-ai/DuplexJev-32B-Para

Finetuned
(97)
this model

Space using adventists-ai/DuplexJev-32B-Para 1

Collection including adventists-ai/DuplexJev-32B-Para

Paper for adventists-ai/DuplexJev-32B-Para