Research map / Vision-language and multimodal / Cross-modal learning
OGP-Net: Optical Guidance Meets Pixel-Level Contrastive Distillation for Robust Multi-Modal and Missing Modality Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OGP-Net, an RGB-IR semantic segmentation network that distills pixel-level knowledge from a frozen RGB DeepLabV3+ teacher via feature exchange, multi-view pixel contrastive learning, and a gated spectral fusion unit, targeting both multi-modal and missing-modality (IR-only) settings.
Problem
RGB-IR semantic segmentation must handle both multi-modal inputs and missing-modality (IR-only) inputs, which prior work usually treats as separate problems; forced feature alignment across large modality gaps can cause negative transfer.
Approach
Shares convolutional filters but uses modality-specific batch norm with channel/spatial feature exchange to form multi-view feature maps; these are trained with supervised pixel-level contrastive learning and distilled from a frozen RGB teacher (DMC), while DUR distills RGB output predictions into the optical branch (DIST loss) and a Gated Spectral Unit fuses branch outputs without manual tuning.
Key results
- On MSRS multi-modal setting, OGP-Net reaches 69.97 mIoU vs 66.24 for MMANet and 66.71 for CDDFuse, with 14.24M params.
- In missing-modality (IR-only) on MSRS, OGP-Net achieves 66.77 mIoU vs 64.67 for SKD-Net.
- On FMB multi-modal, OGP-Net achieves 54.89 mIoU vs 53.97 for MMANet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OGP-Net for RGB-IR segmentation | OGP-Net · MSRS, RGB-IR | mIoU: 66.24 -> 69.97 | Helped |
| use OGP-Net for IR-only inference | OGP-Net · MSRS, IR only | mIoU: 64.67 -> 66.77 | Helped |
| use OGP-Net for RGB-IR segmentation | OGP-Net · MVSS, RGB-IR | mIoU: 51.33 -> 52.90 | Helped |
| use OGP-Net for IR-only inference | OGP-Net · MVSS, IR only | mIoU: 46.83 -> 47.18 | Helped |
| use OGP-Net for RGB-IR segmentation | OGP-Net · FMB, RGB-IR | mIoU: 53.97 -> 54.89 | Helped |
| use OGP-Net for IR-only inference | OGP-Net · FMB, IR only | mIoU: 48.59 -> 48.89 | Helped |
| use OGP-Net with IR-only inference instead of standard distillation losses | OGP-Net · MSRS, IR only | mIoU: 61.84 -> 66.73 | Helped |
| share both convolutional and batch-normalization layers (ablation) | OGP-Net · MSRS, RGB-IR | mIoU: 69.97 -> 67.23 | Hurt |
| keep convolutional and batch-normalization layers unshared (ablation) | OGP-Net · MSRS, RGB-IR | mIoU: 69.97 -> 69.81 | Hurt |
| share convolutions while keeping batch-normalization layers unshared (ablation) | OGP-Net · MSRS, RGB-IR | mIoU: 69.97 -> 69.52 | Hurt |
| add feature exchange (ablation) | OGP-Net · MSRS, RGB-IR | mIoU: 69.52 -> 69.97 | Helped |
| remove pixel-wise contrastive learning (ablation) | OGP-Net · MSRS, RGB-IR | mIoU: 69.97 -> 68.98 | Hurt |
Surprising findings
- Sharing both convolutional and batch norm layers in the encoder drops performance (67.23 vs 69.97 mIoU), while unshared convs with unshared BN gives more parameters but only 69.81.
- Vanilla KD (KL, CWD) from the RGB teacher hurts IR-only performance below the IR baseline, attributed to forced feature alignment.
Limitations
- Sparsity regularization on BN scaling factors had to be disabled during training due to instability with pixel-level distillation.
- Feature exchange and multi-view design rely on a frozen pre-trained RGB teacher, requiring a two-step training pipeline.
Experimental setup
Datasets: MSRS MVSS FMB
Benchmarks: MSRS MVSS FMB
Metrics: mIoU parameter count
Baselines: TarDAL CDDFuse DDFM C.E.N MMANet DisOptNet SKD-Net MFNet RGB2TIR ASAPNet TIR ControlNet DeepLabV3+ KL
Builds on: DeepLabV3+ DIST Channel Exchanging Network CIRKD Gated Spectral Unit
Closest papers by meaning
- Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation (CVPR 2025 · similarity 0.87)
- UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation (NeurIPS 2025 · similarity 0.86)
- Multi-Modal Object Re-identification via Sparse Mixture-of-Experts (ICML 2025 · similarity 0.85)
- OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation (NeurIPS 2025 · similarity 0.85)
- Multi-Modal Contrastive Masked Autoencoders: A Two-Stage Progressive Pre-training Approach for RGBD Datasets (CVPR 2025 · similarity 0.84)
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.