Research map / Vision-language and multimodal / Multimodal LLMs
From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs
Boyong Wu, Sanghwan Kim, Zeynep Akata
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| project vision features through the adapter into the LLM space | LLaVA-1.5 (CLIP, DINOv2, SigLIP) · ADE20K linear probe | mIoU: DINOv2 adapter 42.90 vs encoder baseline above peak 44.50 | Hurt |
| pass image tokens through LLM layers | LLaVA-1.5 CLIP · ADE20K linear probe | mIoU: adapter 33.22 -> peak 40.74 | Improved |
| pass image tokens through LLM layers | LLaVA-1.5 DINOv2 · ADE20K linear probe | mIoU: adapter 42.90 -> peak 44.50; peak stays below encoder (-1.08) | Mixed |
| block attention to correctly classified tokens (ablation) | LLaVA-1.5 CLIP · ADE20K conflicting classes | segmentation quality | Hurt |
| block attention to incorrectly classified tokens (ablation) | LLaVA-1.5 CLIP · ADE20K conflicting classes | segmentation quality | Improved |
| add bidirectional attention among image tokens (ablation) | LLaVA-1.5 CLIP · ADE20K patch 0 | pixel accuracy: 0.6195 -> 0.7630 | Improved |
| add bidirectional attention among image tokens (ablation) | LLaVA-1.5 CLIP · ADE20K linear probe | peak mIoU: 40.74 -> 41.16 | Improved |
| add bidirectional attention among image tokens (ablation) | LLaVA-1.5 DINOv2 · ADE20K linear probe | peak mIoU: 44.50 -> 44.82 | Improved |
| add bidirectional attention among image tokens (ablation) | LLaVA-1.5 SigLIP · ADE20K linear probe | peak mIoU: 35.39 -> 41.26 | Improved |
| add bidirectional attention among image tokens (ablation) | LLaVA-1.5 DINOv2 · VQA benchmarks | accuracy: GQA 62.1 -> 60.5; MMBench 57.7 -> 55.3; VizWiz 51.4 -> 45.8 | Hurt |
| add bidirectional attention among image tokens (ablation) | LLaVA-1.5 CLIP · VQA benchmarks | accuracy: GQA 62.6 -> 62.7; MMBench 66.4 -> 65.5 | No clear effect |
| add bidirectional attention among image tokens (ablation) | LLaVA-1.5 SigLIP · VQA benchmarks | accuracy: MMBench 63.7 -> 66.9; VizWiz 58.2 -> 53.3 | Mixed |
Closest papers by meaning
- Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.90)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.89)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.89)
- SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories (CVPR 2025 · similarity 0.89)
- Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.89)
- A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
