Research map / Vision-language and multimodal / Multimodal LLMs

Explore How to Inject Beneficial Noise in MLLMs

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MuNG, a multimodal noise generator that injects learned, cross-attention-conditioned beneficial noise into frozen MLLM features (between alignment layer and LLM decoder) to act as a parameter-efficient fine-tuning method for Qwen2.5-VL and LLaVA-1.5.

Problem

Existing fine-tuning methods (full fine-tuning, LoRA, DoRA, MBT) for MLLMs either overfit or are single-modality oriented and ignore cross-modal heterogeneity, while full fine-tuning is costly.

Approach

Reformulate MLLM inference via variational inference and learn a Gaussian noise distribution whose mean and variance are produced by a cross-attention generator over visual, text and answer features; the sampled noise is added to visual features before the frozen LLM decoder, framed as positive-incentive noise that suppresses irrelevant semantics.

Key results

• On Qwen2.5-VL-3B, MuNG (0.67% params) reaches average 70.62 vs. 70.41 for LoRA and 68.01 for full fine-tuning.
• On Qwen2.5-VL-7B, MuNG (1.83% params) reaches average 80.51 vs. 80.03 for LoRA and 79.75 for full fine-tuning.
• On LLaVA-1.5-7B, MuNG (2.78% params, combined with LoRA adapters) reaches average 59.3 vs. 60.2 for DoRA and 59.1 for full fine-tuning, so it does not beat DoRA on average.

Surprising findings

• Replacing the cross-attention noise generator with plain Gaussian noise still gives a gain over no noise (71.48 vs 70.49 avg), which the authors interpret as the informative guidance, not randomness, mattering most.
• Freezing all components and training only MuNG performs significantly worse on LLaVA, requiring additional LoRA adapters on the LLM decoder.

Limitations

• Performance drops notably when the LLM decoder is not multimodally pretrained and only MuNG is trained, requiring extra LoRA adapters.
• On LLaVA-1.5-7B, MuNG average (59.3) is below DoRA (60.2), and small score differences on a single run are not analyzed for variance.
• Evaluation relies on an LLM judge (InternLM2.5-1.8B-Chat) and the authors note they could not perfectly reproduce leaderboard numbers.

Experimental setup

Datasets: MMPR-v1.1 LLaVA-Instruct-150K

Benchmarks: MME ScienceQA MM-Vet MMStar POPE SEEDBench MMMU GQA VisWiz TextVQA

Metrics: accuracy average score

Baselines: Full-FT LoRA DoRA MoRA MBT

Builds on: positive-incentive noise LoRA DoRA MoRA Multimodal Bottleneck Transformer

Closest papers by meaning

• Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning (ICML 2025 · similarity 0.91)
• Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning (ICML 2025 · similarity 0.89)
• MokA: Multimodal Low-Rank Adaptation for MLLMs (NeurIPS 2025 · similarity 0.89)
• Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
• Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization (CVPR 2025 · similarity 0.88)
• On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.88)