DuplexJev-4B-Para (v1.4)
π Paper: arXiv:2610.02638 Β· Code: github.com/adventists-ai/duplexjev Β· Release note: hearing the whole clip
The small DuplexJev, now with live turn-taking on par with the 32B model: it knows when the user has finished.
v1.4 (2026-11-04) replaces v1.3 (2026-10-22). Earlier versions stay available as revisions
v1.3,v1.2,v1.1andv1.0(--revision v1.3in vLLM,revision="v1.3"infrom_pretrained).Needs
duplexjev-vllm>=0.4(orduplexjev>=0.7). v1.4 hears the whole audio clip (the Qwen3-ASR encoder emits 6.5 audio tokens per second, not 6.25); the checkpoint sets"audio_full_length": trueinconfig.jsonand the new plugin reads it. Older checkpoints keep their old behaviour.
Architecture: Qwen3-ASR-0.6B encoder + connector + Qwen3-4B, 4.2 B parameters, bf16, ~10 GB GPU memory.
What changed
- Retrained in one pass from v1.1 on the complete data mix (everything v1.2 and v1.3 added, Japanese included), on the true audio length, instead of stacking one more stage on v1.3. Adds the emotion corpora and transcription fixes from DuplexJev-32B-Para v1.4 / v1.5.
- Turn-taking stage: the live end-of-turn data behind DuplexJev-4B-Para-LiveTurn (the assistant's previous message as context, decisions right after a short pause, natural two-speaker English, background speech and noise) plus accent data, with the full mix replayed (a rank-64 LoRA on the LLM, merged here; the connector keeps training).
Results (v1.3 β v1.4, same evaluation code and items; each model with its own audio-length setting)
End of turn
| v1.3 | v1.4 | |
|---|---|---|
| Smart Turn v3.2 test set, accuracy (P(finished) β₯ 0.5, n = 8,749) | 71.0 | 92.9 |
| Smart Turn v3.2 test set, AUC | 0.811 | 0.977 |
| LiveKit eot-bench, English false cut-offs at a 300 ms budget | 25.8 % | 21.8 % |
| LiveKit eot-bench, Chinese false cut-offs at a 300 ms budget | 42.9 % | 39.6 % |
| LiveKit eot-bench, English false cut-offs at 600 ms | 13.0 % | 11.6 % |
eot-bench numbers use audio plus the assistant's previous message, as the Pipecat / LiveKit plugins send it (our fast setting). Smart Turn v3.2 on the same English data: 35.2 % at 300 ms.
Emotion
| v1.3 | v1.4 | |
|---|---|---|
| LISTEN, emotion from voice (LISTEN-EM) | 25.6 | 30.9 |
| LISTEN, neutral text (LISTEN-NT) | 44.5 | 49.9 |
| ESD Chinese / English (held-out speakers) | 90.8 / 81.0 | 95.0 / 83.5 |
| CREMA-D | 81.0 | 84.0 |
Stress and pauses
| v1.3 | v1.4 | |
|---|---|---|
| StressPresso | 80.7 | 87.6 |
| StressTest | 80.3 | 81.7 |
| DEBATE: which word is stressed / does the pause change the meaning | 87.7 / 79.5 | 91.8 / 84.1 |
| Stressed word, Chinese / English (unseen sentences) | 85.1 / 87.2 | 88.0 / 89.8 |
Non-verbal sounds and speaking style
| v1.3 | v1.4 | |
|---|---|---|
| Cough (VocalSound / NonverbalTTS) | 89.4 / 83.4 | 94.5 / 91.4 |
| Sigh (VocalSound / NonverbalSpeech-38K) | 90.5 / 90.2 | 93.9 / 93.3 |
| Laughter in Chinese conversation (MagicData, held-out) | 75.0 | 91.7 |
| Speaking style (EARS / Expresso) | 68.2 / 94.7 | 72.3 / 98.3 |
Serve with vLLM
pip install "vllm[audio]>=0.29" "duplexjev-vllm>=0.4"
vllm serve adventists-ai/DuplexJev-4B-Para --max-model-len 4096
# previous version: add --revision v1.3
Decision prompts are the same as for DuplexJev-4B-Turn; see examples/vllm_client.py. Always offer a "none" option for stress and non-verbal questions. For
live end of turn in a voice agent, use pipecat-duplexjev or livekit-plugins-duplexjev, which also send the
assistant's previous message. For conversation, send the user audio as the user turn (<|audio|>) with ordinary chat
history and sample as usual.
Limitations
- For transcription in Japanese, Korean, Vietnamese, Thai and Arabic the 32B model (DuplexJev-32B-Para) is much stronger.
- Asked about itself, it may describe abilities (seeing images, speaking) that this checkpoint does not have on its own: it takes speech and text and replies in text.
- Speaker judgments are coarse and must not be used to identify or verify people; emotion outputs must not be used to make decisions about individuals.
License
CC BY-NC 4.0, non-commercial use only.
- Non-commercial data:
- Stress: DEBATE (github.com/SmileHnu/DEBATE), Stress-17K and TinyStress-15K (slprl);
- Expresso (Meta), EARS (Meta) and NonVerbalSpeech-38K, all CC BY-NC 4.0;
- ESD (research only); SmoothConv (CC BY-NC);
- Conversation replay: user turns from wilzzzz/paralinguistic_dialogues (no licence stated on the dataset card; used for non-commercial research), SmoothConv (CC BY-NC), AISHELL-1 (Apache-2.0), LibriSpeech (CC BY 4.0); target replies generated by Qwen3-4B (Apache-2.0); spoken chat questions synthesised with CosyVoice2 (Apache-2.0).
- Added in v1.2 (non-commercial or share-alike): Emilia JA/KO (CC BY-NC 4.0); VIVOS (CC BY-NC-SA 4.0); a licensed Mandarin conversational corpus with paralinguistic annotation (MagicData, purchased licence; audio not redistributed).
- Added in v1.2 (other): Emilia-YODAS JA/KO (CC BY 4.0), YODAS2 (CC BY 3.0), Zeroth-Korean (CC BY 4.0), FLEURS train split (CC BY 4.0; the test split was never used), Common Voice 17 ar/ru (CC0). Labels were re-checked, and for unlabelled YODAS2 clips produced, with Qwen3-ASR-0.6B (Apache-2.0); continuation targets were written by Qwen3-32B (Apache-2.0).
- Added in v1.3: Japanese speech from Emilia JA (CC BY-NC 4.0), Emilia-YODAS JA (CC BY 4.0), Common Voice ja (CC0) and other Japanese speech data, used for non-commercial research; every label re-checked with Qwen3-ASR-0.6B (Apache-2.0), continuation targets written by Qwen3-32B (Apache-2.0). The held-out Emilia JA shard used for testing was never trained on.
- Added in v1.4 (non-commercial or share-alike): EmotionTalk (CC BY-NC-SA 4.0); DailyTalk (CC BY-SA 4.0); accents from KeSpeech (non-commercial research use), English Dialects (CC BY-SA 4.0) and EdAcc (CC BY-SA 4.0).
- Added in v1.4 (other): CSEMOTIONS (Apache-2.0), EMNS (Apache-2.0), JL-Corpus (CC0), Emozionalmente (CC BY 4.0), SUBESCO (CC BY 4.0); FLEURS dev split (CC BY 4.0; the test split was never used); accents from VCTK (CC BY 4.0) and common-accent from Common Voice (CC0 audio); turn-taking: otoSpeech-141h (CC BY 4.0) with DualTurn labels and the Smart Turn v3.2 training set (pipecat-ai, CC BY 4.0; the test set was never trained on).
- Other data:
- VocalSound (CC BY-SA 4.0), NonverbalTTS (audio from VoxCeleb and Expresso);
- VoxCeleb1 (CC BY 4.0), AISHELL-1 (Apache-2.0), LibriSpeech (CC BY 4.0);
- ARC (CC BY-SA 4.0), OpenBookQA, CommonsenseQA (MIT).
- Base models keep their licences: Qwen3-4B and Qwen3-ASR-0.6B are Apache-2.0 (Alibaba Cloud); the Qwen3-4B licence is included. The LLM weights are Qwen3-4B with LoRAs merged in.
Citation
@misc{jin2026duplexjev,
title = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
author = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
year = {2026},
eprint = {2610.02638},
archivePrefix = {arXiv},
note = {Submitted to IEEE ICASSP 2027}
}
Built by AIιδΈ΄ζ΄Ύ / Adventists.ai. Claude (Anthropic) assisted with code.
- Downloads last month
- 448