Research map / Vision-language and multimodal / Multimodal LLMs

Language-guided Frequency Modulation for Large Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Language-guided Frequency Modulation (LFM), a parameter-free plug-in that modulates grouped vision-encoder features in the Fourier domain with a cosine-similarity emphasis map conditioned on text, improving LVLMs across many benchmarks.

Problem

Existing LVLM visual refinement operates mainly in the spatial domain and does not explicitly separate high-frequency local details from low-frequency global context, which limits fine-grained, language-aligned control of visual tokens.

Approach

Fourier-transform grouped multi-layer ViT features, then reweight frequency components by a cosine similarity between each frequency representation and the language embedding (emphasis map), and inverse-transform back into the spatial tokens before an MLP projector; the modulation adds no trainable parameters beyond the projector.

Key results

• LFM with Vicuna-13B and LLaVA-1.5 data reaches MMBench 74.8 and MME-P 1631 at 384 resolution, compared to 74.4 and 1579 for Dense Connector with Vicuna-13B on 1.2M+1.5M data.
• Integrating LFM into Qwen2.5-VL-7B raises MMStar from 63.9 to 66.1 and MathVista from 68.2 to 69.8.
• In the interaction-strategy ablation, frequency-domain cosine modulation gives the best gain (MMStar +2.18, MMBench +3.18 over no-interaction baseline) with no trainable parameters beyond the projector.

Surprising findings

• Frequency-domain interaction outperforms spatial-domain cross-attention with extra trainable parameters, and the parameter-free cosine modulation performs best in the ablation.

Limitations

• Gains on some benchmarks are modest or mixed, e.g. MM-Vet and MMMU drop for some configurations versus baselines in Table 2 and Table 3.
• Hyperparameters (alpha schedule, beta, grouping) are tuned on MMBench and may not generalize without tuning.
• Mechanistic claim that frequency bands map to global versus local semantics is supported mainly by illustrative figures rather than systematic analysis.

Experimental setup

Datasets: LLaVA-1.5 pretraining data LLaVA-1.5 instruction tuning data Mini-Gemini data InternVL-1.2 SFT data

Benchmarks: GQA SQA-I TextVQA MME MMBench MMBench v1.1 MME-RealWorld MMStar MM-Vet LLaVA-Bench-In-the-Wild MathVista MMMU

Metrics: accuracy score

Baselines: LLaVA-1.5 MGM Dense Connector Qwen2.5-VL InternVL2.5 ShareGPT4V LLaMA-VID VILA CuMo MM1

Builds on: LLaVA-1.5 Mini-Gemini (MGM) Dense Connector CLIP

Closest papers by meaning

• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.89)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.89)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.89)
• Language-Instructed Vision Embeddings for Controllable and Generalizable Perception (ICLR 2026 · similarity 0.89)
• From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.88)