Research map / Large language models / Language, safety and interpretability
DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representations
Chao-Hong Tan, Qian Chen, Wen Wang, Chong Deng, Qinglin Zhang, Luyao Cheng, Hai Yu, Xin Zhang, Xiang Lyu, Tianyu Zhao, Chong Zhang, Yukun Ma and 5 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add continuous speech encoder (Whisper features) for speech input (ablation) | DrVoice-Small · Llama Questions S2T | score: 62.33 -> 72.33 | Improved |
| add continuous speech encoder (ablation) | DrVoice-Small · Llama Questions S2M (text) | score: 61.67 -> 68.67 | Improved |
| add continuous speech encoder (ablation) | DrVoice-Small · Llama Questions T2T | score: 74.00 -> 75.33 | No clear effect |
| add Speech Refined Head (SRH) (ablation) | DrVoice-Small · Llama Questions S2M (text) | score: 21.67 -> 38.33 | Improved |
| add Speech Refined Head (SRH) (ablation) | DrVoice-Small · Llama Questions T2M (text) | score: 45.22 -> 59.33 | Improved |
| add Speech Refined Head (SRH) (ablation) | DrVoice-Small · Llama Questions T2T | score: 73.00 -> 73.33 | No clear effect |
| add SRH pretraining (ablation) | DrVoice-Small · Llama Questions S2M (text) | score: 38.33 -> 61.67 | Improved |
| add CoM-Mixing training (ablation) | DrVoice-Small · Llama Questions S2M (text) | score: 58.00 -> 68.67 | Improved |
| raise grouping factor from 1 to 5 (ablation) | DrVoice-Small · Grouping sweep S2T | score: 55.67 -> 63.33 | Improved |
| raise grouping factor from 5 to 7 (ablation) | DrVoice-Small · Grouping sweep S2M (text/speech) | score: 37.67 / 28.00 -> 36.00 / 16.67 | Hurt |
| use grouping factor 5 instead of 1 (ablation) | DrVoice-Small · Training GPU hours | GPU hours: nearly 50% reduction | Improved |
| add Stage 2 low-LR fine-tuning after merging (ablation) | DrVoice 7B · VoiceBench (Inhouse v2, MagpiePro, InfGen) | average: 70.19 -> 74.73 | Improved |
Closest papers by meaning
- Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.88)
- CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations (NeurIPS 2024 · similarity 0.87)
- NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction (ICML 2025 · similarity 0.87)
- CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching (NeurIPS 2025 · similarity 0.86)
- Scaling Speech-Text Pre-training with Synthetic Interleaved Data (ICLR 2025 · similarity 0.86)
- PolyVoice: Language Models for Speech to Speech Translation (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
