Research map / Vision-language and multimodal / Multimodal LLMs
F-LMM: Grounding Frozen Large Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Grounds frozen off-the-shelf LMMs for segmentation by translating word-to-image attention maps into masks with a small CNN decoder and a SAM-based refiner, preserving conversational ability without fine-tuning the LMM.
Problem
Grounding LMMs fine-tuned with segmentation tokens lose general conversational and instruction-following ability, while grounded-conversation data is costly to annotate.
Approach
Keep the LMM frozen and exploit word-to-image attention maps already present in its self-attention as segmentation priors; a tiny trainable U-Net maps stacked attention maps to mask logits, refined by a SAM mask head that uses box and language cues, plus a linear keyword selector for choosing grounded words.
Key results
- F-LMM (LLaVA-1.6-7B / MGM-HD-7B) reaches about 66-67 PNG recall and 75-76 RefCOCO val cIoU while retaining general QA scores (e.g., MME ~1500, MMBench ~65-70) where grounding LMMs score near zero.
- On reasoning segmentation (ReasonSeg), F-LMM achieves 46.2 gIoU on test, above LISA's 36.8 without training on such data.
- With visual CoT prompting, F-LMM (DeepSeek-VL-7B) improves VisCoT benchmark DocVQA from 43.2 to 53.8 and outperforms VisCoT-7B on the same benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Freeze the LMM and train only a light mask head (CNN decoder + SAM refiner) instead of fine-tuning LMM weights with [SEG] tokens | grounding LMMs · MMBench / LLaVA-In-the-Wild / MMVet | instruction-following and QA scores | Helped |
| Ground frozen LMM attention maps (no fine-tuning, no grounded instruction data) with CNN decoder + SAM refiner | DeepSeekVL-1.3B · PNG All | recall | No clear effect |
| Add SAM mask refiner on top of mask decoder output (ablation) | F-LMM mask decoder · PNG All | recall: +12.6 | Helped |
| Add box and text prompts to the mask refiner (ablation) | F-LMM mask refiner · PNG All | recall: +1.5 | Helped |
| Use SAM ViT-L instead of ViT-B as mask refiner (ablation) | F-LMM mask refiner · PNG All | recall: +1.9 | Helped |
| Average (vs max) merging of multi-word attention maps (ablation) | F-LMM mask decoder · PNG | recall: +0.7 | Helped |
| Use attention maps from all layers vs late layers only (ablation) | F-LMM mask decoder · PNG | recall | Helped |
| Normalize inputs to the mask decoder (ablation) | F-LMM mask decoder · PNG | recall: +0.5 | Helped |
| Replace SpaCy parser with trained linear keyword selector (ablation) | keyword selection for grounded conversation · PNG | F1: +25.0 | Helped |
| Visual chain-of-thought prompting (localise then zoom in) (ablation) | F-LMM (DeepSeekVL-7B) · VisCoT benchmark (DocVQA) | accuracy: +10.6 | Helped |
| Visual chain-of-thought prompting (ablation) | F-LMM (DeepSeekVL-7B) · POPE | accuracy: +1.0 | Helped |
| Scaling frozen LMM backbone size/quality (chat ability) | F-LMM across 10 LMM backbones · PNG and RES | grounding rank vs chat rank | Helped |
Surprising findings
- Existing grounding LMMs (GLaMM, LISA, PixelLM) score zero or near zero on MMBench and MME, failing to answer yes/no questions with single words.
- Attention maps from late transformer layers perform worst for segmentation, while using all layers gives the best results.
Limitations
- Inherits biases and limitations of the frozen underlying LMMs.
- Only vision-language modality studied; no video, audio, or 3D.
- Experiments limited to LMMs up to 8B parameters due to compute.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg PNG
Benchmarks: MME MMBench MMVet LLaVA-In-the-Wild RefCOCO RefCOCO+ RefCOCOg PNG ReasonSeg GCG VisCoT POPE
Metrics: cIoU recall gIoU METEOR mIoU accuracy F1
Baselines: PixelLM LISA GLaMM GroundHog LLaVA-G LaSagnA GSVA PerceptionGPT VisCoT OMG-LLaVA BuboGPT KOSMOS-2
Builds on: SAM LLaVA-1.5 DeepSeek-VL LISA GLaMM
Closest papers by meaning
- GLaMM: Pixel Grounding Large Multimodal Model (CVPR 2024 · similarity 0.92)
- NExT-Chat: An LMM for Chat, Detection and Segmentation (ICML 2024 · similarity 0.89)
- VideoGLaMM : A Large Multimodal Model for Pixel-Level Visual Grounding in Videos (CVPR 2025 · similarity 0.89)
- MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs (NeurIPS 2024 · similarity 0.89)
- GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.89)
- GROUNDHOG: Grounding Large Language Models to Holistic Segmentation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.