Research map / Video / Video segmentation and tracking
End-to-End Video Semantic Segmentation in Adverse Weather using Fusion Blocks and Temporal-Spatial Teacher-Student Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an end-to-end, optical-flow-free domain-adaptive video semantic segmentation method for adverse weather, combining a deformable-conv fusion block, temporal-spatial teacher-student learning, and temporal weather degradation augmentation.
Problem
Video semantic segmentation under unsupervised domain adaptation from synthetic to real adverse-weather videos fails because pretrained optical flow and pseudo-labels become unreliable in fog, rain, snow, and night.
Approach
Replace pretrained optical flow with a learned fusion block (deformable-convolution offset layers) that matches and merges features from adjacent frames, trained end-to-end with segmentation losses; a temporal teacher (full frames) and a spatial teacher (high-resolution crop) jointly supervise a student that sees only a cropped current frame plus the full adjacent frame, with temporal weather degr
Key results
- Adapting VIPER to MVSS yields 25.4% mIoU, a 4.3-point gain over the best prior method (TPS, 21.1 vs 20.2 baseline context in Table 1 second best).
- Adapting Synthia to MVSS yields 33.0% mIoU, a 5.8-point improvement over the strongest prior method (TPS at 27.8%).
- Ablation on VIPER to MVSS: Accel without optical flow drops to 18.6 mIoU; adding the fusion block recovers to 21.7 and the full model reaches 25.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed method instead of RDA | Video domain adaptation from VIPER to MVSS · VIPER → MVSS (adverse weather) | mIoU (%): 4.3% | Helped |
| Use the proposed method instead of TPS | Video domain adaptation from Synthia to MVSS · Synthia → MVSS (adverse weather) | mIoU (%): 5.8% | Helped |
| Use the proposed method instead of SFC | Video domain adaptation from VIPER to Cityscapes-Seq · VIPER → Cityscapes-Seq (ideal conditions) | mIoU | Hurt |
| Use the proposed method instead of SFC | Video domain adaptation from Synthia to Cityscapes-Seq · Synthia → Cityscapes-Seq (ideal conditions) | mIoU | Hurt |
| Remove pretrained optical flow, with no fusion block or teachers (ablation) | Accel baseline on VIPER → MVSS · VIPER → MVSS (adverse weather) | mIoU (%) | Hurt |
| Add a fusion block (ablation) | Accel (NOOF) · VIPER → MVSS (adverse weather) | mIoU (%): 18.6 -> 21.7 | Helped |
| Add a temporal teacher (ablation) | Accel + fusion block · VIPER → MVSS (adverse weather) | mIoU (%): 21.7 -> 23.8 | Helped |
| Add a spatial teacher (ablation) | Fusion block + temporal teacher · VIPER → MVSS (adverse weather) | mIoU (%): 23.8 -> 24.3 | Helped |
| Add temporal weather degradation augmentations (ablation) | Fusion block + temporal and spatial teachers · VIPER → MVSS (adverse weather) | mIoU (%): 24.3 -> 25.4 | Helped |
Surprising findings
- Video-based UDA methods outperform image-based ones under ideal conditions but not under adverse weather, indicating failure to exploit temporal information when pseudo-labels and optical flow are unreliable.
Limitations
- The paper's limitations section is marked NA in the checklist; no explicit limitations are discussed.
- On Synthia to Cityscapes-Seq (Table 6), the method (51.0) trails SFC (55.3) and TPS (53.8), so gains are not universal under ideal conditions.
Experimental setup
Datasets: VIPER Synthia MVSS Cityscapes-Seq
Benchmarks: MVSS Cityscapes-Seq
Metrics: IoU mIoU
Baselines: Source-only AdvEnt FDA RDA DA-VSN SFC TPS MoDA VAT-VST
Builds on: Accel TPS DA-VSN Mean Teacher-style EMA teacher DeepLabV2
Closest papers by meaning
- Semantic Segmentation in Multiple Adverse Weather Conditions with Domain Knowledge Retention (AAAI 2024 · similarity 0.91)
- Parsing All Adverse Scenes: Severity-Aware Semantic Segmentation with Mask-Enhanced Cross-Domain Consistency (AAAI 2024 · similarity 0.90)
- Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal (CVPR 2024 · similarity 0.88)
- ERF: A Benchmark Dataset for Robust Semantic Segmentation Under Extreme Rainfall Conditions (AAAI 2025 · similarity 0.87)
- Semantic Segmentation on Raindrop Degraded Images Using Two-Stage Dual Teacher-Student Learning (AAAI 2025 · similarity 0.86)
- Continuous Adverse Weather Removal via Degradation-Aware Distillation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.