Research map / Vision-language and multimodal / Cross-modal learning
From Natural Alignment to Conditional Controllability in Multimodal Dialogue
Zeyu Jin, Songtao Zhou, Haoyu Wang, Minghao Tian, Kaifeng Yun, Zhuo Chen, Xiaoyu Qin, Jia Jia
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on MM-Dia | Higgs-Audio-V2 · MM-Dia Test (Description control) | WER: 31.3 -> 4.5 | Improved |
| fine-tune on MM-Dia | Higgs-Audio-V2 · MM-Dia Test (Description control) | cp-WER: 104.8 -> 33.8 | Improved |
| fine-tune on MM-Dia | Higgs-Audio-V2 · MM-Dia Test (Description control) | sa-SIM: 0.48 -> 0.45 | Hurt |
| fine-tune on MM-Dia | Higgs-Audio-V2 · MM-Dia Test (Description control) | Human-MOS quality: 3.580 -> 4.440 | Improved |
| fine-tune on MM-Dia | Dia · MM-Dia Test (Description control) | WER: 19.991 -> 29.071 | Hurt |
| fine-tune on MM-Dia | Dia · MM-Dia Test (Description control) | Human-MOS quality: 2.410 -> 2.890 | Improved |
| use cascaded GPT style prompts instead of end-to-end HarmoniVox | Higgs-Audio-SFT · MM-Dia-Bench | WER: 21.223 -> 5.793 | Improved |
| provide style cues implicitly via visual context instead of explicit description | Higgs-Audio-SFT · MM-Dia-Bench | Gemini-as-Judge similarity: 4.640 -> 3.022 | Hurt |
| calibrate SRT subtitles (ablation) | SRT subtitles · Subtitle timestamp | timestamp accuracy: 0.179 -> 0.857 | Improved |
| use buffer b=2 instead of b=1 (ablation) | Qwen 7B segmentation · Dialogue segmentation | F1 overlap: 0.771 -> 0.866 | Improved |
| increase buffer from b=1 to b=5 (ablation) | Qwen 72B segmentation · Dialogue segmentation | F1 overlap: 0.947 -> 0.979 | Improved |
| use Gemini instead of Qwen for non-verbal sound annotation | annotation · MM-Dia annotation | completeness: 1.25 -> 4.66 | Improved |
Closest papers by meaning
- U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation (CVPR 2026 · similarity 0.88)
- Structure-Aware Multimodal Sequential Learning for Visual Dialog (AAAI 2024 · similarity 0.88)
- TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation (NeurIPS 2025 · similarity 0.87)
- CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations (NeurIPS 2024 · similarity 0.86)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.86)
- CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
