Research map / Vision-language and multimodal / Cross-modal learning
Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a symmetric parameter-efficient fine-tuning framework for RGB+X semantic segmentation with modality-aware prompting/adaptation, dynamic sparse cross-modality fusion, and masked modality self-teaching, evaluated on six multimodal segmentation datasets.
Problem
Multimodal (RGB+X) semantic segmentation methods either fully fine-tune costly dual backbones or use asymmetric PEFT designs that treat the X modality as a secondary prompt, and attention-based fusion introduces cross-modal noise.
Approach
Freeze a pre-trained Swin Transformer and treat RGB and X symmetrically: a shared-then-projected prompt generator and modality-specific adapters adapt the backbone to each modality; a deformable-style sparse fusion selects dominant points across modalities; and a masked self-teaching loss transfers correct dual-modality predictions to single-modality heads.
Key results
- Swin-L variant reaches 59.9% mIoU on NYUDv2 and 55.0% on SUN RGB-D, surpassing prior PEFT and matching/exceeding FFT methods such as GeminiFusion with 16.1M trainable parameters vs 369.2M.
- On PST900 the method outperforms the previous best PEFT method by 7.2% mIoU and the best FFT method by 2.1%.
- On MCubeS, Swin-B reaches 53.8% (RGB-NIR), 54.5% (RGB-D) and 53.7% (RGB-AoLP), setting SOTA on those settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the symmetric PEFT framework instead of the strongest listed PEFT baseline | Ours · PST900 | mIoU (%): 81.5 -> 88.7 | Helped |
| Use the symmetric PEFT framework instead of the strongest listed PEFT baseline | Ours · NYUDv2 | mIoU (%): 54.9 -> 59.9 | Helped |
| Use the symmetric PEFT framework instead of the strongest listed PEFT baseline | Ours · SUN RGB-D | mIoU (%): 52.3 -> 55.0 | Helped |
| Use the symmetric PEFT framework instead of the strongest listed PEFT baseline | Ours · MFNet | mIoU (%): 57.7 -> 59.9 | Helped |
| Use the symmetric PEFT framework instead of the strongest listed PEFT baseline | Ours · MCubeS RGB-D | mIoU (%): 52.7 -> 54.5 | Helped |
| Use the symmetric PEFT framework instead of the strongest listed PEFT baseline | Ours · DeLiVER RGB-E | mIoU (%): 57.8 -> 58.4 | Helped |
| Add prompt generation (ablation) | Ours on Swin-B · NYUDv2 | mIoU (%): 57.7 (-1.3) | Helped |
| Add prompt generation (ablation) | Ours on Swin-B · MFNet | mIoU (%): 57.5 (-2.4) | Helped |
| Add modality-specific adapters (ablation) | Ours on Swin-B · NYUDv2 | mIoU (%): 56.2 (-2.8) | Helped |
| Add modality-specific adapters (ablation) | Ours on Swin-B · MFNet | mIoU (%): 57.8 (-2.1) | Helped |
| Add DSCF instead of directly adding modality features (ablation) | Ours on Swin-B · NYUDv2 | mIoU (%): 57.1 (-1.9) | Helped |
| Add DSCF instead of directly adding modality features (ablation) | Ours on Swin-B · MFNet | mIoU (%): 56.9 (-3.0) | Helped |
Surprising findings
- Modality-specific adapters are critical: removing them drops mIoU by 2.8% on NYUDv2, while shared adapters at the same width lose 1.0%.
Limitations
- Not explored for vision-language tasks; language modalities are left for future work.
- Swin-L underperforms Swin-B on MFNet, attributed to the small dataset and nighttime images.
Experimental setup
Datasets: NYUDv2 SUN RGB-D MFNet PST900 MCubeS DeLiVER
Benchmarks: NYUDv2 SUN RGB-D MFNet PST900 MCubeS DeLiVER
Metrics: mIoU trainable parameters FPS
Baselines: CMNeXt CMX GeminiFusion DPLNet GoPT OMNIVORE StitchFusion Any2Seg TokenFusion DFormer
Builds on: Swin Transformer deformable attention DPLNet GoPT CMNeXt
Closest papers by meaning
- Prompting Multi-Modal Image Segmentation with Semantic Grouping (AAAI 2024 · similarity 0.89)
- UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation (NeurIPS 2025 · similarity 0.88)
- Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation (NeurIPS 2024 · similarity 0.88)
- OGP-Net: Optical Guidance Meets Pixel-Level Contrastive Distillation for Robust Multi-Modal and Missing Modality Segmentation (AAAI 2025 · similarity 0.87)
- OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation (NeurIPS 2025 · similarity 0.87)
- SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.