Research map / Video / Video segmentation and tracking
Unleashing Multispectral Video's Potential in Semantic Segmentation: A Semi-supervised Viewpoint and New UAV-View Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MVUAV, a UAV oblique-view RGB-thermal video segmentation benchmark, and SemiMV, a semi-supervised baseline using cross-collaborative consistency learning and denoised memory reads.
Problem
Multispectral (RGB-thermal) video semantic segmentation suffers from few datasets, which are all eye-level views, and sparse frame annotations; semi-supervised methods for this setting are unexplored.
Approach
Mutual pseudo-supervision between RGB and thermal streams (cross-collaborative consistency learning, with cross-modal fusion) plus a KL-divergence-based reliability map that denoises prototype memory reads from unlabeled past frames.
Key results
- SemiMV reaches 43.04 mIoU on MVSeg at 1/4 labeled partition vs 36.88 for supervised RGB-T baseline and 39.27 for CPS (RGB).
- On MVUAV at 1/16 labeled split, SemiMV reaches 15.10 mIoU vs 11.28 for supervised RGB-T baseline and 13.36 for UniMatch.
- Cross-collaborative consistency (C3L) alone raises the 1/4 MVSeg mIoU from 36.88 to 40.73; removing cross-modal collaboration drops it to 36.67.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SemiMV instead of SupOnly (RGBT) | MVSS segmentation · MVSeg 1/16, 1/8, 1/4, 1/2 | mIoU: +2.22%, +5.67%, +6.16%, +5.98% | Helped |
| Use SemiMV instead of SupOnly (RGBT) | MVSS segmentation · MVUAV 1/16, 1/8, 1/4, 1/2 | mIoU: +3.82%, +5.16%, +5.21%, +4.63% | Helped |
| Add a thermal infrared branch to labeled RGB training (ablation) | SemiMV supervised baseline · MVSeg 1/4 | mIoU: 35.79% → 36.88% (+1.09%) | Helped |
| Add C3L to RGB-Thermal training (ablation) | SemiMV · MVSeg 1/4 | mIoU: 36.88% → 40.73% (+3.85%) | Helped |
| Add denoised memory read (DMR) (ablation) | SemiMV with C3L · MVSeg 1/4 | mIoU: +1.66% mIoU | Helped |
| Add Dual-C3L on updated query features (ablation) | SemiMV with DMR · MVSeg 1/4 | mIoU: 42.39% → 43.04% | Helped |
| Add cross supervision instead of self-generated pseudo-labels (ablation) | C3L · MVSeg 1/4 | mIoU: 39.12% → 40.73% (1.61%) | Helped |
| Add cross-modal collaboration (ablation) | C3L with cross pseudo supervision · MVSeg 1/4 | mIoU: 36.67 → 40.73 | Helped |
| Add temporal aggregation from unlabeled past frames (ablation) | SemiMV · MVSeg 1/4 | mIoU: 40.73 → 41.47; 40.73 → 41.85 | Helped |
| Denoise memory features using the reliability estimation strategy (ablation) | DMR-STM and DMR-Proto · MVSeg 1/4 | mIoU: 41.47 → 42.25; 41.85 → 42.39 | Helped |
| Use DMR-Proto rather than DMR-STM (ablation) | SemiMV temporal integration · MVSeg 1/4 | mIoU: 42.25; 42.39 | Mixed |
Surprising findings
- Naive consistency learning between RGB and thermal streams without cross-modal collaboration decreases performance (36.67 vs 36.88 baseline).
- Pixel-wise pseudo labels from one modality alone can be incomplete, causing error accumulation in cross-pseudo supervision.
Limitations
- Gains on the harder MVUAV benchmark remain modest in absolute mIoU (values around 15-32).
- Cross-modal fusion block is borrowed from prior work rather than newly designed.
Experimental setup
Datasets: MVUAV MVSeg ImageNet
Benchmarks: MVSeg MVUAV
Metrics: mIoU
Baselines: MT CCT CPS UniMatch Accel IFR MVNet SupOnly (RGB) SupOnly (RGBT)
Builds on: DeepLabv3+ CPS MVNet CMF IFR
Closest papers by meaning
- High Temporal Consistency through Semantic Similarity Propagation in Semi-Supervised Video Semantic Segmentation for Autonomous Flight (CVPR 2025 · similarity 0.87)
- Multiview Aerial Visual RECognition (MAVREC): Can Multi-view Improve Aerial Visual Perception? (CVPR 2024 · similarity 0.86)
- Leader360V: A Large-scale, Real-world 360 Video Dataset for Multi-task Learning in Diverse Environment (NeurIPS 2025 · similarity 0.85)
- PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation (CVPR 2026 · similarity 0.85)
- ROS-SAM: High-Quality Interactive Segmentation for Remote Sensing Moving Object (CVPR 2025 · similarity 0.85)
- Weakly Supervised Semantic Segmentation for Driving Scenes (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.