DuplexJev-32B-Para (v1.5)
📄 Paper: arXiv:2610.02638 · Code: github.com/adventists-ai/duplexjev · Release notes: hearing the whole clip (v1.5) · Japanese as a third language · weak languages · Research notes: hearing stress · keeping a speech-decision LLM able to talk
🗣️ Now with a voice: DuplexJev-32B-Para is the ears of Sophon-Realtime v1, our open full-duplex voice model that hears your tone and speaks with one. Released: Tag and Flash, one GPU each.
The DuplexJev model that hears the most beyond the words. On top of everything DuplexJev-32B-Turn and DuplexJev-32B-Stress do (spoken QA, gender, emotion, turn-taking, stress and pauses), it also hears laughter, breathing, coughs and sighs, notices how someone is talking (whispering, shouting, very fast or slow), and can tell, roughly as a person would, whether two clips are the same speaker. Mandarin and English. Since v1.1 it also holds a normal conversation, multi-turn included. v1.2 adds accent judgments (English accents and Mandarin dialect regions) and much sharper end-of-turn detection right after the user pauses: on LiveKit's eot-bench it now beats Smart Turn v3.2 in English and Chinese. v1.3 hears stress much better: on the public StressTest benchmark it goes from 72 to 85, on StressPresso's meaning questions from 69 to 89, and on choosing the meaning a stressed word implies (DEBATE, held-out) from 85 to 93, with the other decisions about the same. v1.4 understands far more languages: Japanese error goes from 33 % to 12 % (FLEURS) and from 38 % to 11 % on conversational Japanese, Korean and Vietnamese error falls to a quarter to a third of v1.3, Thai, Arabic and Russian error by 30–40 %, and Mandarin conversational paralinguistics (laughter, breath, drawl, stutter …) are much sharper. v1.5 hears the whole clip: earlier versions (like every Qwen3-ASR-based checkpoint we trained) dropped the last ~4 % of every 30 s audio chunk, because the connector assumed 6.25 audio tokens per second while the encoder emits 6.5. v1.5 is retrained on the true length, which sharpens the decisions that depend on the end of an utterance: end-of-turn (Smart Turn test set 93.3 → 94.4, Chinese 91.4 → 94.6; LiveKit eot-bench English false cut-offs at 300 ms 20.4 → 17.7 %), stress (StressTest 81.7 → 85.3), and emotion in natural speech (LISTEN emotion 32.4 → 33.2, LISTEN non-verbal 48.8 → 51.6), with spoken QA up a point as well.
v1.5 (2026-11-03) replaces v1.4 (2026-10-22): audio is read at its true length (
"audio_full_length": trueinconfig.json). Useduplexjev-vllm>=0.4/duplexjev>=0.7, which read this flag; older versions still load the model but cut the last ~4 % of each 30 s chunk, as before. Earlier versions stay available as revisionsv1.4,v1.3,v1.2,v1.1andv1.0(--revision v1.4in vLLM,revision="v1.4"infrom_pretrained).
DuplexJev reads typed, closed-set decisions from speech as a single-token distribution of the LLM: zero decode steps, many questions about the same clip in parallel. This repository is the complete model (Qwen3-ASR-0.6B encoder
- connector + the language model of Qwen3-VL-32B, 33.0 B parameters, bf16; one 80 GB GPU).
Which 32B to pick. This one, for everything above.
What changed
Starting from DuplexJev-32B-Turn we trained the connector and a new rank-16 LoRA on the LLM (merged here) for 4,000 steps. The mix is the 32B-Turn mix plus:
| added data | what it teaches |
|---|---|
| stress / pause detection (as in 32B-Stress: DEBATE, Stress-17K, TinyStress-15K, Expresso, neutral negatives) | which word is stressed, or none; where the pause is |
| speaker pairs (VoxCeleb1 dev, AISHELL-1, LibriSpeech, two-speaker mixtures) | same speaker or not; is this person in the clip |
| non-verbal sounds (VocalSound, NonVerbalSpeech-38K, NonverbalTTS, Expresso laughing) | is there laughter / breathing / a cough / a sigh / sniffing / throat clearing / a sneeze / crying …; negatives are cut from the same recordings away from the event |
| speaking style (EARS, Expresso whisper) | whispering, loud, fast, slow, high or low pitch, normal |
| spoken multiple-choice replay (ARC, OpenBookQA and CommonsenseQA train splits read by TTS) | keeps "answer the content" from being crowded out by "listen to the voice" |
| v1.1: conversation replay (spoken and typed user turns, up to 6 earlier turns as history; target = the original LLM's own reply) and in-house chat items | normal, full-length replies; trained with per-row loss averaging so long replies do not swamp the one-token decisions |
| v1.2: accents (VCTK, English Dialects, EdAcc, Common Voice accent labels, KeSpeech; held-out speakers kept for testing) | which accent: 19 English accents and 9 Mandarin dialect regions; native speaker or not; standard Mandarin or not; which English region |
| v1.2: end of turn right after a pause (otoSpeech-141h cut 0.2 s into each pause, with DualTurn labels; Smart Turn v3.2 training set) | finished vs. still talking / hesitating when the user has just gone quiet, the way a live voice agent has to decide |
| v1.3: the stress data re-balanced (more "which meaning does this stress imply" items, weight raised) and the gender data weighted ×2; no new corpora | choosing the meaning a stressed word implies; keeping gender where it was |
| v1.4: weak-language speech (ja, ko, vi, th, ar, ru: Emilia, Emilia-YODAS, YODAS2, Zeroth-Korean, VIVOS, FLEURS train, Common Voice; 162 k, labels re-checked with Qwen3-ASR-0.6B) + continuation (36 k) | transcribe and understand these languages |
| v1.4: Japanese speech (Emilia JA, Emilia-YODAS JA, Common Voice ja and other Japanese speech data; 358 k, re-checked) + Japanese continuation (20 k) | Japanese close to Chinese and English |
| v1.4: Mandarin conversational paralinguistics (licensed MagicData corpus, training speakers only; 42 k questions) | laughter, laughing while talking, breath, lip smack, sniff, cough, stressed word, drawl, which character is drawn out, stutter, sudden noise, unnatural pause, sentence emotion |
| v1.4: more emotion classes (surprise, fear, anxiety, excitement, encouragement, apologetic, playful) from CSEMOTIONS, EmotionTalk, DailyTalk, JL-Corpus, EMNS, Emozionalmente, SUBESCO | emotions beyond the basic four |
| v1.5: the whole v1.4 mix again, now with audio tokens at the encoder's true length (6.5 per second); transcription items with native-language instructions and leading / trailing silence (FLEURS dev split, the v1.4 weak-language speech); a light replay of all emotion data | the end of every clip is heard; always transcribe when asked to; emotion kept sharp |
v1.4 was trained from v1.3 in four short stages (2,000 + 2,000 + 1,500 + 1,000 steps; connector + a new rank-16 LoRA each time, merged here), replaying the full v1.3 mix in every stage; the last two stages raise the stress, non-verbal, emotion and speaking-style replay ×1.5 each.
v1.5 was trained from v1.4 in two stages (2,000 + 1,500 steps; connector + a new rank-16 LoRA each time, merged here) with the audio-token count computed from the encoder's real output length.
v1.5 vs v1.4
| v1.4 | v1.5 | |
|---|---|---|
| qa100 / ZJU-ML (vLLM, client wording) | 97 / 90 | 98 / 91 |
| gender / emotion (vLLM) | 92.1 / 89.2 | 92.6 / 90.0 |
| Easy-Turn* (vLLM) | 94.1 | 95.4 |
| Smart Turn v3.2 test set, all / Chinese (threshold 0.5) | 93.3 / 91.4 | 94.4 / 94.6 |
| eot-bench, English false cut-offs at 300 / 600 ms (with the assistant's previous message, fast mode) | 20.4 / 8.9% | 17.7 / 7.0% |
| eot-bench, Chinese, same setting | 42.3 / 13.8% | 41.0 / 13.3% |
| StressTest / StressPresso (which meaning the stress implies) | 81.7 / 88.6 | 85.3 / 89.1 |
| DEBATE: stressed word / pause position | 89.8 / 79.8 | 90.7 / 81.4 |
| LISTEN (EACL 2026) emotion / non-verbal, audio only | 32.4 / 48.8 | 33.2 / 51.6 |
| EmotionTalk / CSEMOTIONS emotion (test splits)† | 62.6 / 89.8 | 68.5 / 94.2 |
Same harness for both columns (stress, LISTEN and emotion rows: single-token readout, PyTorch, % correct). † In-domain: the training splits of these corpora are in the replay data.
v1.4 vs v1.3
Languages (FLEURS test, 150 utterances per language, median word / character error, lower is better):
| ja | ko | vi | th | ar | ru | zh | en | es | fr | de | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| v1.3 | 32.7 | 31.8 | 41.7 | 29.1 | 27.0 | 10.6 | 3.8 | 1.5 | 4.4 | 8.5 | 5.3 |
| v1.4 | 11.6 | 7.6 | 14.6 | 18.0 | 19.4 | 6.9 | 3.8 | 0.0 | 3.5 | 7.0 | 5.3 |
| encoder ceiling (Qwen3-ASR-0.6B) | 7.0 | 2.7 | 7.8 | 5.1 | 16.7 | 7.8 | 2.0 | 1.3 | 3.4 | 6.2 | 5.6 |
Japanese, conversational (held-out Emilia JA shard) / Common Voice ja test, character error: v1.3 38.2 / 33.3 → v1.4 11.1 / 13.0 (encoder 8.3 / 10.8).
Mandarin conversational paralinguistics (licensed MagicData corpus, held-out speakers, accuracy %; yes/no unless noted):
| breath | lip smack | laughter | laughing while talking | sniff | cough | drawl | stressed word (4-way) | emotion (4-way) | |
|---|---|---|---|---|---|---|---|---|---|
| v1.4 | 91.7 | 85.8 | 89.3 | 87.8 | 87.5 | 71.4 | 79.4 | 85.3 | 70.7 |
(v1.3 was not trained on this corpus; the 4B-Para v1.1, also untrained on it, scores 50–66 on the yes/no rows.)
Accent, same-speaker and turn-taking (eot-bench, Smart Turn) were not re-run for v1.4; the numbers in the next sections are v1.3's.
Results (v1.3 and earlier)
| 32B-Turn | 32B-Para v1.1 | 32B-Para v1.2 | 32B-Para v1.3 | |
|---|---|---|---|---|
| qa100 / ZJU-ML | 96 / 87 | 96 / 88 | 95 / 87 | 96 / 87 |
| gender / emotion | 91.5 / 91.1 | 92.9 / 91.5 | 93.9 / 90.1 | 93.8 / 92.1 |
| VoiceBench OBQA‡ / MMSU | 85.5 / 72.1 | 94.5 / 75.5 | 94.3 / 75.0 | 94.1 / 75.7 |
| Easy-Turn* | 95.2 | 94.8 | 95.5 | 94.4 |
| which word is stressed, "none" offered: DEBATE (zh) / StressPresso (en) | 35 / 1 | 93 / 96 | 92 / 91 | 95 / 96 |
| neutral speech → "no word is stressed": AISHELL-1 / LibriSpeech | 92 / 50 | 100 / 99 | 99 / — | 98 / 83 |
| meaning from stress: DEBATE held-out / StressTest | 35 / 52 | 85 / 76 | 85 / 72 | 93 / 85 |
| StressPresso, which meaning the stress implies (English) | — | — | 69 | 89 |
| laughter / breathing / cough / sigh (AUC, NonverbalTTS test, real speech, unseen) | — | 0.88 / 0.81 / 0.85 / 0.80 | 0.87 / 0.81 / 0.84 / 0.75 | ≈ v1.2† |
| laughter / breathing in natural Mandarin conversation (AUC, unseen third-party sample) | — | 0.83 / 0.65 | 0.80 / 0.56 | ≈ v1.2† |
| speaking style, 7 classes (EARS held-out speakers) | — | 82% | 79% | 80% |
| same speaker? EER, VoxCeleb1 random / hard pairs (our harness) | — | 28.2% / 39.6% | 31.2% / 42.3% | 29.1% / 38.4%◊ |
| accent, 4 choices: English (19 accents) / Mandarin (9 dialect regions)§ | — | 39.6 / 25.7 | 56.0 / 48.2 | — |
| **native English speaker? / standard Mandarin?**§ | — | 84.8 / 46.6 | 91.1 / 91.3 | — |
| LISTEN (EACL 2026), mean of the 4 audio-only conditions, full set | — | — | 38.4 | 38.9 |
Paper protocol (single-token readout, PyTorch), % correct unless marked; AUC 0.5 = chance.
Through vLLM with duplexjev-vllm in the client wording, v1.3: qa100 97, ZJU-ML 90, gender 92.1, emotion 91.8,
Easy-Turn 94.5 (v1.2: 98, 90, 93.6, 89.8, 95.5; v1.1: 97, 91, 93.2, 90.0, 94.8; qa100 and ZJU-ML have 100 items each).
"—": not re-run for that version.
† v1.3 non-verbal sounds were checked by accuracy on the same test sets (NonverbalTTS, NonVerbalSpeech-38K, VocalSound, Expresso, MagicData): within a few points of v1.2 either way; AUC not re-run.
◊ Same harness re-run for v1.2: 30.7% / 38.2% (the v1.2 column is from an earlier run).
Accents for v1.3 were checked only on the cross-corpus set: standard Mandarin or not 88.8 (v1.2 93.1, 160 items), Mandarin dialect 4-way 61.3 (60.0); English accents stay near chance cross-corpus for both versions.
* Easy-Turn is in-domain (its training split is in the mix).
§ Accent test items come from the same corpora as the training items (different speakers); chance is 25% / 50%.
‡ OpenBookQA's train split is in the replay data, so the OBQA gain is partly in-domain; MMSU is unrelated to it.
For reference, an AudioSet AST tagger scores 0.71 / 0.75 / 0.85 / 0.76 on the same four sounds.
Speaker: a dedicated speaker-verification model reaches ~1% EER on VoxCeleb1; this model is meant to have a rough, human-like sense of "same voice or not", not to identify people.
Turn-taking benchmarks (v1.2 / v1.3; v1.5 in the table above)
LiveKit eot-bench (validation set, decision 0.2 s into each pause, official metrics). False cut-offs when the agent may wait at most 300 / 600 ms (lower is better):
| English | Chinese | |
|---|---|---|
| LiveKit Turn Detector v1 | 9.9 / 4.5% | 20.6 / 8.7% |
| DuplexJev-32B-Para v1.2, with the assistant's previous message (plugin default, full eot-bench run) | 20.0 / 9.4% | 37.7 / 12.2% |
DuplexJev-32B-Para v1.2, with the assistant's previous message, readout="act" (fast mode) |
18.4 / 7.9% | 37.3 / 12.3% |
| DuplexJev-32B-Para v1.2, audio only (fast mode) | 23.5 / 11.1% | 40.0 / 12.8% |
| DuplexJev-32B-Para v1.3, with the assistant's previous message (fast mode) | 20.3 / 8.8% | 35.5 / 12.3% |
| DuplexJev-32B-Para v1.3, audio only (fast mode) | 24.7 / 11.2% | 38.8 / 12.0% |
| DuplexJev-32B-Turn, audio only (fast mode) | 29.8 / 13.9% | 44.9 / 14.5% |
| Smart Turn v3.2 | 35.2 / 14.8% | 42.9 / 13.2% |
eot-bench audio is not in our training data. "With the assistant's previous message" puts
The assistant had just said: "..." before the audio; the Pipecat and LiveKit plugins send it
(pipecat-duplexjev>=0.3, livekit-plugins-duplexjev>=0.2, gateway duplexjev>=0.6). Plugin default: the turn
question, P(finished) + P(backchannel); run with every eot-bench decision point (adapter
benchmarks/turn_detection/eot_bench/duplexjev_ctx_adapter.py on GitHub). readout="act" (pipecat-duplexjev>=0.4,
livekit-plugins-duplexjev>=0.3): question "What should the assistant do now?", P(reply now) + P(short
acknowledgement). Rows marked fast mode ask once per pause, at the 0.2 s score point.
Pipecat Smart Turn v3.2 test set (accuracy at threshold 0.5): v1.2 English 94.0% (Smart Turn v3.2: 94.7%), Chinese 91.1% (90.5%); v1.3 94.2% / 91.4%; DuplexJev-32B-Turn 65.2% / 84.6%. The Smart Turn v3.2 training set is in our training mix, so this is an in-distribution result.
Serve with vLLM
pip install "vllm[audio]>=0.29" "duplexjev-vllm>=0.4"
vllm serve adventists-ai/DuplexJev-32B-Para --max-model-len 4096
duplexjev-vllm>=0.4 reads "audio_full_length": true from config.json and feeds the whole clip; DJ_FULL_LEN=0
forces the old truncated length, DJ_FULL_LEN=1 the full length for any checkpoint.
Prompt format and client are the same as for DuplexJev-32B-Turn;
see examples/vllm_client.py. Examples:
The user said: <|audio|>
Is there laughter in this audio?
Options:
A. Yes
B. No
Answer with the letter only.
用户说:<|audio|>
除了说话,这段音频里还有哪种声音?
选项:
A. 叹气
B. 咳嗽
C. 笑声
D. 都没有
请只回答选项字母。
Always offer a "none" option for stress and non-verbal questions; the model was trained with it.
Limitations
- Accent and same-speaker scores were not re-measured for v1.4 / v1.5.
- Thai and Arabic are still 2–3× the encoder's own error.
- Non-verbal training audio is mostly acted (radio drama, animation, studio recordings); in natural conversation the scores are lower (see the table).
- Breathing in natural Mandarin conversation is weak (AUC 0.56 in v1.2, 0.65 in v1.1, 0.78 in v1.0); acted / read speech is much better.
- Accent answers describe how someone sounds, not where they are from or who they are; accents are often mixed or ambiguous. Do not use them to make decisions about people.
- Asked about itself, it may describe abilities (seeing images, speaking) that this checkpoint does not have on its own: it takes speech and text and replies in text.
- Speaker judgments are coarse (EER 18–40%) and must not be used to identify or verify people.
- Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.
License
CC BY-NC 4.0, non-commercial use only.
- Non-commercial data:
- Stress: DEBATE (github.com/SmileHnu/DEBATE), Stress-17K and TinyStress-15K (slprl);
- Expresso (Meta), EARS (Meta) and NonVerbalSpeech-38K, all CC BY-NC 4.0;
- ESD (research only); SmoothConv (CC BY-NC).
- Accents (v1.2): KeSpeech (non-commercial research use).
- Conversation replay (v1.1): user turns from wilzzzz/paralinguistic_dialogues (no licence stated on the dataset card; used for non-commercial research), SmoothConv (CC BY-NC), AISHELL-1 (Apache-2.0), LibriSpeech (CC BY 4.0); target replies generated by the base LLM (Apache-2.0); spoken chat questions synthesised with CosyVoice2 (Apache-2.0).
- Added in v1.4 (non-commercial or share-alike): Emilia JA/KO (CC BY-NC 4.0); VIVOS (CC BY-NC-SA 4.0); EmotionTalk (CC BY-NC-SA 4.0); DailyTalk (CC BY-SA 4.0); a licensed Mandarin conversational corpus with paralinguistic annotation (MagicData, purchased licence; audio not redistributed); other Japanese speech data used for non-commercial research.
- Added in v1.4 (other): Emilia-YODAS JA/KO (CC BY 4.0), YODAS2 (CC BY 3.0), Zeroth-Korean (CC BY 4.0), FLEURS train split (CC BY 4.0; the test split was never used), Common Voice 17 ja/ar/ru (CC0), CSEMOTIONS (Apache-2.0), EMNS (Apache-2.0), JL-Corpus (CC0), Emozionalmente (CC BY 4.0), SUBESCO (CC BY 4.0). Labels were re-checked, and for unlabelled YODAS2 clips produced, with Qwen3-ASR-0.6B (Apache-2.0); continuation targets were written by Qwen3-32B (Apache-2.0).
- Added in v1.5: FLEURS dev split (CC BY 4.0; the test split was never used).
- Other data:
- VocalSound (CC BY-SA 4.0), NonverbalTTS (audio from VoxCeleb and Expresso);
- VoxCeleb1 (CC BY 4.0), AISHELL-1 (Apache-2.0), LibriSpeech (CC BY 4.0);
- ARC (CC BY-SA 4.0), OpenBookQA, CommonsenseQA (MIT).
- Accents (v1.2): VCTK (CC BY 4.0), English Dialects (CC BY-SA 4.0), EdAcc (CC BY-SA 4.0), common-accent from Common Voice (CC0 audio).
- Turn-taking (v1.2): otoSpeech-141h (CC BY 4.0) with DualTurn labels; Smart Turn v3.2 training set (pipecat-ai, CC BY 4.0).
- Base models keep their licences: Qwen3-VL-32B-Instruct and Qwen3-ASR-0.6B are Apache-2.0 (Alibaba Cloud).
- The LLM weights are the language model of Qwen3-VL-32B-Instruct with LoRAs merged in; the vision encoder is not included.
Citation
@misc{jin2026duplexjev,
title = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
author = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
year = {2026},
eprint = {2610.02638},
archivePrefix = {arXiv},
note = {Submitted to IEEE ICASSP 2027}
}
Built by Adventists.ai. Claude (Anthropic) assisted with code.
- Downloads last month
- 346
Model tree for adventists-ai/DuplexJev-32B-Para
Base model
Qwen/Qwen3-ASR-0.6B