Research map / Vision-language and multimodal / Multimodal LLMs

Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SMT, an MLLM (Whisper encoder + Q-Former adapter + GemmaX2-28-9B LLM) that fuses synthetic speech from CosyVoice2 with text for translation, using a COMET-based self-evolution loop that keeps only speech samples that improve translation.

Problem

Image-guided multimodal machine translation is limited by scarce multilingual image-text pairs and has shown limited generalization to general MT datasets; the paper asks whether speech, which is abundant and naturally aligned with text, can improve translation quality.

Approach

Use TTS-synthesized speech as an auxiliary modality fused with text in an MLLM, and iteratively self-train on synthetic speech samples labeled positive when speech input raises COMET over text-only translation.

Key results

• On Multi30K eng→deu Test2016, SMT-9B reaches 47.0 BLEU / 87.8 COMET, above the prior best IMAGE (45.3) and text-only DeepSeek-V3.1 (44.2).
• On FLORES-200 eng→xx (27 targets), SMT-9B averages 40.4 spBLEU / 89.5 COMET... with the reported average of 39.7 for baseline text-only, indicating average gains across 108 directions reported as SOTA.
• Self-evolution improves low-resource COMET (khm, lao, mya) with round 3 giving about +1.9, +2.0, and +1.7 COMET; ablation shows w/o SE drops FLORES average to 33.0 spBLEU vs 34.3 for SMT-9B.

Surprising findings

• Synthetic speech input achieves comparable or slightly better S2TT performance than authentic speech on CoVoST-2, attributed to absence of background noise.
• Text-only LoRA baseline yields much of the gain on Multi30K, suggesting part of the improvement is from fine-tuning rather than speech.

Limitations

• Language coverage is bounded by the languages supported by the TTS model (28 languages in the model).
• Long inputs (e.g., WMT24++) produce noisy synthetic speech (word omissions, durations over 30s), degrading performance.
• Self-evolution gains diminish after the first round; benefits are modest on high-resource languages.

Experimental setup

Datasets: FLEURS Common Voice 19 Multi30K CoVoST-2 FLORES-200 WMT24++

Benchmarks: Multi30K FLORES-200 WMT24++ CoVoST-2

Metrics: BLEU spBLEU COMET

Baselines: DeepSeek-V3.1 Gemma3-27B-it NLLB-moe-54B Qwen3-Next-80B-A3B Soul-Mix RG-MMT-EDC WRA-guided ConsQA-MMT VALHALLA Bridge DreamLLM IMAGE

Builds on: Whisper Q-Former CosyVoice2 GemmaX2-28-9B COMET

Closest papers by meaning

• A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.87)
• High-Fidelity Simultaneous Speech-To-Speech Translation (ICML 2025 · similarity 0.87)
• Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.86)
• Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.86)
• DiM$^3$: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging (NeurIPS 2026 · similarity 0.86)
• NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.86)