Research map / Vision-language and multimodal / Multimodal LLMs

CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CREMA, a modular, parameter-efficient video-language reasoning framework that adds per-modality LoRA-based query adapters on a frozen BLIP-2 Q-Former, fuses auxiliary modality queries with a self-gated module, and trains modalities sequentially with adaptive early exit.

Problem

Multimodal LLMs that reason over video plus extra modalities (audio, 3D, depth, flow, touch, thermal) are costly to adapt, usually require separate large Q-Formers per modality, and fixed modality sets limit flexibility; adding modalities also inflates LLM input length.

Approach

Keep the pre-trained vision-language Q-Former frozen and add lightweight modality-specific multi-query adapters (LoRA rank 64, learnable queries, linear projectors), fuse the auxiliary modality queries via a self-gated module to keep LLM token count small, and train modalities sequentially within each minibatch with gradient-based early exit per modality.

Key results

• On MUSIC-AVQA, CREMA with V, A, F, D, N reaches 81.7% average accuracy versus 78.1% for BLIP-2 with V, A, F, using about 38M trainable parameters vs. 324M for BLIP-2.
• On SQA3D, CREMA with V, P, D, N achieves 54.6% average accuracy, above 3D-LLM's 52.3% (V, P) with 38M vs. 434M trainable parameters.
• On NExT-QA, CREMA with V, F, D, N reaches 73.9% average accuracy vs. 73.3% for BLIP-2 with 28M vs. 432M trainable parameters; adaptive early exit cuts training computation by 25-60%.

Measured effects reported in the paper

Surprising findings

• Adding modalities to BLIP-2 often degrades performance (e.g., MUSIC-AVQA V+A 78.4 vs. V 78.9), while CREMA improves consistently with more modalities.
• Zero-shot, OneLLM's accuracy on SQA3D drops when combining video and point cloud (39.4 with V alone vs. 37.9 with V+P).

Limitations

• Auxiliary modalities (depth, flow, normals) are extracted by frozen pre-trained estimators, so quality depends on those experts.
• Evaluation on a limited set of tasks and a single LLM backbone family at the main scale; stronger MLLMs left for future work.
• Zero-shot results on SQA3D and MUSIC-AVQA use only two modalities at a time with the fusion module bypassed.

Experimental setup

Datasets: SQA3D MUSIC-AVQA NExT-QA Touch&Go Thermal-IM PerceptionTest VGGSound AudioCaps 3D-LLM data

Benchmarks: SQA3D MUSIC-AVQA NExT-QA TouchQA ThermalQA PerceptionTest VGGSound

Metrics: accuracy trainable parameters GFLOPs training memory

Baselines: BLIP-2 3D-LLM LAVISH AVQA SeViLA OneLLM X-InstructBLIP CMC MCAN ClipBERT ScanQA LLaMA-VQA

Builds on: BLIP-2 Q-Former LoRA SeViLA X-InstructBLIP

Closest papers by meaning

• A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.90)
• MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.89)
• MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.89)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.89)
• Incentivizing Versatile Video Reasoning in MLLMs via Data-Efficient Reinforcement Learning (CVPR 2026 · similarity 0.88)
• Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering (ICML 2025 · similarity 0.88)