Research map / Vision-language and multimodal / Cross-modal learning

ModaVerse: Efficiently Transforming Modalities with LLMs

CVPR 2024 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 14 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ModaVerse, an MLLM that trains linear input adaptors plus a LoRA on Vicuna and uses the LLM as an agent emitting meta-responses that invoke frozen text-to-image/video/audio models, aligned via instruction tuning at the language level.

Problem

Existing MLLMs that understand and generate multiple modalities require complex multi-stage latent-feature alignment with large data and compute, while LLM-as-agent pipelines lack training and are inconsistent.

Approach

Combine adaptor training on the input side (linear projections from ImageBind features into the LLM space) with an LLM-as-agent output side, aligning LLM outputs to text-to-x model inputs via instruction-following I/O alignment at the natural language level, avoiding output-side projection layers and generative model training.

Key results

• Text-to-image FID of 11.24 on COCO caption, comparable to SD (11.21) and NExT-GPT (11.28).
• Text-to-video FID 13.35 and CLIPSIM 0.3014 on MSR-VTT, close to NExT-GPT (13.04, 0.3085).
• Uses under 2% of the data volume of Emu and BLIP-2 (2M vs 82M/129M) and a single training stage, on 4 GPUs for 20 hours per Table 1.

Measured effects reported in the paper

Surprising findings

• Without language cues in the input, the model tends to produce irrelevant outputs.

Limitations

• Does not preserve original input layout; image editing produces entirely new images.
• Does not outperform all SOTA methods on the six benchmarks.
• Relies on language instructions; produces irrelevant outputs without language cues.

Experimental setup

Datasets: LLaVA instruction data VideoChat InstructBLIP COCO caption AudioCaps MSR-VTT generative model training datasets

Benchmarks: COCO caption AudioCaps MSR-VTT

Metrics: FID B@4 METEOR CIDEr FD IS SPIDEr CLIPSIM

Baselines: Emu CoDi NExT-GPT OFA GLIDE CogVideo Stable Diffusion BLIP-2 GIT mPLUG-2

Builds on: LLaVA BLIP-2 ImageBind HuggingGPT NExT-GPT LoRA

Closest papers by meaning

• Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.92)
• SAE-V: Interpreting Multimodal Models for Enhanced Alignment (ICML 2025 · similarity 0.90)
• DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.89)
• L-Man: A Large Multi-modal Model Unifying Human-centric Tasks (AAAI 2025 · similarity 0.89)
• HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.88)
• HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding (ICML 2025 · similarity 0.88)