Research map / Vision-language and multimodal / CLIP and zero-shot
Prompting Multi-Modal Image Segmentation with Semantic Grouping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GoPT, a frozen-backbone prompt tuning framework with class-aware uni-modal and alignment-induced cross-modal grouping prompters for RGB-D, RGB-T, and RGB-SAR segmentation, training under 1% of parameters.
Problem
Multi-modal image segmentation must integrate cross-modal common information while retaining modality-specific patterns; full fine-tuning of RGB-pretrained foundation models is parameter-heavy and transfers poorly with scarce multi-modal data.
Approach
Freeze an MAE-pretrained ViT and insert grouping prompters: a class-aware uni-modal prompter groups auxiliary-modality tokens via Gumbel-Softmax hard assignment to learned class tokens, and an alignment-induced cross-modal prompter aggregates these class-aware prompts back into the RGB stream via cross-attention, with class tokens inherited across layers.
Key results
- GoPT (L) reaches 54.3% mIoU on NYUDv2 and 52.3% mIoU on SUN RGB-D, outperforming the RGB foundation segmenter SETR by about 4.8 and 3.4 points.
- GoPT (B) achieves 57.4% mIoU on MFNet and GoPT (L) achieves 81.5% mIoU on PST900, the best among compared RGB-T methods.
- GoPT (B) reaches 72.9% OA, 69.3% AA and 60.7% Kappa on WHU-OS, beating the runner-up MIFNet by 2.8% AA and 0.9% Kappa; the method trains only 0.97M parameters versus 112.59M for full fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GoPT prompt tuning for RGB-D segmentation | MAE ViT-B · NYUDv2 | mIoU: 49.3 -> 54.1 | Helped |
| Add GoPT prompt tuning for RGB-D segmentation | MAE ViT-L · SUN RGB-D | mIoU: 48.9 -> 52.3 | Helped |
| Add GoPT prompt tuning for RGB-T segmentation | MAE ViT-B · MFNet | mIoU: +1.2 | Helped |
| Add GoPT prompt tuning for RGB-T segmentation | MAE ViT-L · PST900 | mIoU: +1.0 | Helped |
| Add GoPT prompt tuning for RGB-SAR segmentation | MAE ViT-B · WHU-OS | AA: +2.8% | Helped |
| Replace GoPT with shallow VPT-style prompts (ablation) | GoPT · NYUDv2 | mIoU: 54.1 -> 50.2 | Hurt |
| Replace GoPT with deep VPT-style prompts (ablation) | GoPT · NYUDv2 | mIoU: 54.1 -> 52.8 | Hurt |
| Use shallow grouping prompters (ablation) | GoPT · NYUDv2 | mIoU: 54.1 -> 52.9 | Hurt |
| Remove ACP (equivalent to adding ACP) (ablation) | GoPT · NYUDv2 | mIoU: 53.7 -> 54.1 | Helped |
| Remove CUP (equivalent to adding CUP) (ablation) | GoPT · NYUDv2 | mIoU: 53.9 -> 54.1 | Helped |
| Remove ACP (equivalent to adding ACP) (ablation) | GoPT · MFNet | mIoU: 56.4 -> 57.4 | Helped |
| Remove CUP (equivalent to adding CUP) (ablation) | GoPT · MFNet | mIoU: 56.8 -> 57.4 | Helped |
Surprising findings
- Soft assignment in the grouping prompter performs notably worse than hard assignment, which the authors attribute to feature ambiguity across prompts.
- Adding the proposed prompters to the frozen RGB backbone outperforms full fine-tuning of an auxiliary-branch multi-modal network on NYUDv2, MFNet and WHU-OS.
Limitations
- Evaluated only on a small set of downstream multi-modal benchmarks with a single backbone family (MAE ViT).
- Segmentation head must still be fine-tuned per dataset because class sets differ.
Experimental setup
Datasets: NYUDv2 SUN RGB-D MFNet PST900 WHU-OS
Benchmarks: NYUDv2 SUN RGB-D MFNet PST900 WHU-OS
Metrics: pixel accuracy mean accuracy mIoU overall accuracy average accuracy Kappa coefficient
Baselines: SETR EVP VPT-Shallow VPT-Deep Full fine-tuning (FFT) RSFNet MIFNet CMX-style RGB-D/T fusion methods
Builds on: VPT MAE SETR Gumbel-Softmax
Closest papers by meaning
- Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation (CVPR 2025 · similarity 0.89)
- COMMA: Co-articulated Multi-Modal Learning (AAAI 2024 · similarity 0.87)
- Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach (CVPR 2024 · similarity 0.87)
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.86)
- Prompt-Driven Referring Image Segmentation with Instance Contrasting (CVPR 2024 · similarity 0.86)
- Delving into Multimodal Prompting for Fine-Grained Visual Classification (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.