Research map / Vision-language and multimodal / Multimodal LLMs

Vision-Speech Models: Teaching Speech Models to Converse about Images

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts the open-weight full-duplex speech LLM Moshi into a vision-speech conversational model (MoshiVis) using lightweight gated cross-attention adapters, mixed speechless and speech training, and synthetic spoken visual dialogues.

Problem

Equipping a pretrained real-time speech dialogue model with visual understanding is hard because paired image-speech data is scarce, real-time latency limits compute, and the model must keep general conversational and prosodic abilities while switching between image and non-image topics.

Approach

Freeze Moshi and the PaliGemma image encoder and train only gated cross-attention adapters, using mixed batches of speechless image-text data (with Moshi's text stream) and a small share of speech samples, plus a learned per-token gate that enables switching between visual and general conversation.

Key results

• Training with 0% audio and evaluating in audio gives 38.5% on OCR-VQA, 49.3% on VQAv2 and CIDEr 113 on COCO; adding 1% audio recovers MOSNet from 2.78 to 3.59 (Moshi backbone 3.34).
• With 10% spoken OCR-VQA samples, audio-evaluated accuracy reaches 60.7% versus 36.8% with no audio samples for that task.
• Gated adapters add about 7 ms per inference step on an L4 GPU (51 ms vs. the backbone's roughly 44 ms at conversation start), staying under the 80 ms real-time threshold.

Surprising findings

• Training on speechless (text-only) image data transfers to audio querying with substantial accuracy, even though the text stream is out-of-distribution, and task knowledge transfers better through audio than through text.
• Sharing gate parameters across layers sometimes outperforms per-layer gating while being more parameter-efficient.

Limitations

• Speech-prompted performance trails text-prompted performance, partly due to punctuation and formatting loss in speech output.
• Conversational model is more verbose, lowering CIDEr scores (about 80 vs about 125 when trained on COCO).
• Synthetic dialogue data biases the model toward 'yes' answers, hurting yes/no questions like OCR-VQA.

Experimental setup

Datasets: COCO OCR-VQA VQAv2 PixMo DOCCI PixelProse TallyQA RenderedText DocVQA

Benchmarks: COCO OCR-VQA VQAv2 MMLU

Metrics: CIDEr accuracy VQA accuracy MOSNet latency per inference step

Baselines: PaliGemma stage 3 Moshi backbone cascaded text-speech conversion

Builds on: Moshi PaliGemma Flamingo-style perceptual augmentation Mistral-Nemo

Closest papers by meaning

• VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction (NeurIPS 2025 · similarity 0.90)
• Structure-Aware Multimodal Sequential Learning for Visual Dialog (AAAI 2024 · similarity 0.88)
• Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.88)
• Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.88)
• VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks (NeurIPS 2024 · similarity 0.88)
• MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs (NeurIPS 2024 · similarity 0.88)