Research map / Vision-language and multimodal / Multimodal LLMs
Instruction-Guided Visual Masking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Instruction-guided Visual Masking (IVM), a plug-and-play SAM-based heatmap model trained on a 1M-sample auto- and human-annotated dataset with discriminator-weighted supervised learning, which masks instruction-irrelevant image regions to boost LMMs (GPT-4V, LLaVA-7B) and robot policies.
Problem
Multimodal LMMs often misalign textual instructions with the relevant local image regions and get distracted by instruction-irrelevant content, and existing grounding methods are limited to fixed categories or instruction formats.
Approach
Train a generic instruction-conditioned heatmap model (LLaVA-7B with LoRA plus SAM-H backbone and lightweight generator) on mixed-quality masks, weighting training samples with a learned discriminator (DWSL), and use its heatmaps to mask or crop images before feeding them to any downstream multimodal model without fine-tuning.
Key results
- IVM improves GPT-4V accuracy on V*Bench by 26.2 points over the vanilla 55.0%, surpassing the SEAL specialist.
- IVM-enhanced LLaVA-7B reaches 87.2 on POPE and 1806 on MME, outperforming LLaVA-13B on EgoThink (54.5 vs 55.2 is slightly lower, so only POPE and MME are higher) and POPE/MME.
- LISA-enhanced LLaVA-7B degrades on all benchmarks (e.g., POPE 80.0 vs 85.9 baseline), while IVM improves them.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IVM masking | LLaVA-7B · EgoThink | score: +3.4 | Helped |
| Add IVM masking | LLaVA-7B · POPE | score: +1.3 | Helped |
| Add IVM masking | LLaVA-7B · MME* | score: +58 | Helped |
| Add IVM masking | LLaVA-7B · GQA | score: +0.2 | No clear effect |
| Add IVM masking | LLaVA-7B · SQA | score | No clear effect |
| Add IVM masking | LLaVA-7B · VQAv2 | score: +0.5 | No clear effect |
| Apply LISA masking instead of IVM masking | LLaVA-7B · EgoThink | score: -3.2 | Hurt |
| Apply LISA masking instead of IVM masking | LLaVA-7B · POPE | score: -5.9 | Hurt |
| Apply LISA masking instead of IVM masking | LLaVA-7B · MME* | score: -188 | Hurt |
| Increase machine-annotated training data volume (ablation) | IVM · V*Bench | accuracy improvement | Helped |
| Add human annotations to the training data (ablation) | IVM · V*Bench | accuracy improvement | Helped |
| Use DWSL instead of traditional supervised learning (ablation) | IVM · V*Bench | accuracy improvement | Helped |
Surprising findings
- LISA-based visual masking, despite a 13B model and prompt tuning, consistently degrades LMM performance across all tasks.
- The IVM-enhanced 7B LLaVA outperforms the 13B LLaVA on POPE and MME, yet the 13B model still scores higher on EgoThink.
Limitations
- Adds parameters and inference overhead; heuristic mask deployment (overlay/crop) may be suboptimal and cause distribution shift for LMMs.
- Heatmaps lack exact object contours; failures include missing small targets, misguided targets, and insufficient reasoning.
- DWSL relies on a learned discriminator and a hand-designed weighting function f(x), which may not filter all label errors.
Experimental setup
Datasets: IVM-Mix-1M VG caption Flickr30K VSR OpenImage RefCOCO LLaVA-Instruction-tuning OpenX
Benchmarks: V*Bench EgoThink POPE MME GQA SQA VQAv2 RefCOCO RefCOCO+ RefCOCOg
Metrics: accuracy success rate
Baselines: LISA-Enhanced LLaVA-7B SEAL LLaVA-13B InstructBLIP G-DINO-L
Builds on: LISA Segment Anything Model LLaVA offline imitation learning with mixed-quality data LoRA
Closest papers by meaning
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.89)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.88)
- LLaVA-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (ICLR 2025 · similarity 0.88)
- Improved Baselines with Visual Instruction Tuning (CVPR 2024 · similarity 0.88)
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.88)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.