Research map / Video / Video segmentation and tracking
High Temporal Consistency through Semantic Similarity Propagation in Semi-Supervised Video Semantic Segmentation for Autonomous Flight
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSP, a lightweight video semantic segmentation method that propagates logits across frames via learned similarity-based linear interpolation and global homography alignment, plus a consistency-aware knowledge distillation (KD-SSP) for sparsely labeled aerial videos.
Problem
Semantic segmentation of aerial (UAV) video needs to be accurate and temporally consistent while running in real time onboard, but aerial video datasets are sparsely labeled with few annotated frames.
Approach
Propagate predictions across frames by per-pixel linear interpolation between the current image-model logits and the homography-aligned previous prediction, with weights produced by learnable convolutional similarity layers on feature maps (replacing TFC's cosine similarity); train with a teacher-based knowledge distillation over all frames using optical-flow-consistent teacher predictions.
Key results
- KD-SSP improves TC over the base image model by 12.5% on UAVid and 6.7% on RuralScapes (SSP base training: 13% and 6.7% per text), with higher mIoU (80.63 vs 79.23 on UAVid).
- SSP reaches 92.10 TC on UAVid and 94.06 TC on RuralScapes versus 79.02 and 87.34 for the base image model; KD-SSP runs at 95 FPS on A100 and 29.3 FPS on Jetson Orin.
- KD-SSP outperforms video methods DFF, NetWarp, and TCB variants in mIoU while running at much higher inference speed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SSP prediction propagation to the base image model | Hiera-S + UPerNet · UAVid | TC: 79.02 -> 92.10 | Helped |
| Add SSP prediction propagation to the base image model | Hiera-S + UPerNet · RuralScapes | TC: 87.34 -> 94.06 | Helped |
| Add knowledge distillation to SSP | SSP · UAVid | mIoU: 79.75 -> 80.63 | Helped |
| Add knowledge distillation to SSP | SSP · RuralScapes | mIoU: 64.00 -> 64.56 | Helped |
| Set consistency loss weight λ to 0.2 instead of λ = 0.5 (ablation) | SSP · UAVid | TC: 92.01 -> 90.72 | Hurt |
| Set consistency loss weight λ to 0.35 instead of λ = 0.5 (ablation) | SSP · UAVid | mIoU: 79.87 -> 79.98 | No clear effect |
| Set consistency loss weight λ to 0.65 instead of λ = 0.5 (ablation) | SSP · UAVid | TC: 92.01 -> 92.18 | Helped |
| Replace convolutional similarity with cosine similarity interpolation (ablation) | SSP · UAVid | mIoU: 79.87 -> 79.20 | Hurt |
| Remove global registration alignment (ablation) | SSP · UAVid | TC: 92.01 -> 89.65 | Hurt |
| Add linear interpolation for prediction propagation (ablation) | SSP · UAVid | TC: 88.20 -> 92.01 | Helped |
| Add consistency loss (ablation) | SSP · UAVid | TC: 86.38 -> 92.01 | Helped |
| Train SSP in one step rather than two steps (ablation) | SSP · UAVid | TC: 92.01 -> 91.97 | No clear effect |
Surprising findings
- Removing global registration alignment keeps mIoU essentially unchanged but significantly reduces temporal consistency (92.01 to 89.65 TC on UAVid).
- Replacing the learned convolutional similarity layer with TFC-style cosine similarity lowers both accuracy and consistency on UAVid.
Limitations
- Global homography assumes a planar ground surface, causing alignment errors in complex scenes.
- Evaluation limited to two aerial datasets; teacher and student share architecture family and are trained on sparse labels.
- Speed benchmark excludes the global registration estimation step.
Experimental setup
Datasets: UAVid RuralScapes
Benchmarks: UAVid RuralScapes
Metrics: mIoU temporal consistency (TC) FPS GFLOPs
Baselines: Base image model (Hiera-S + UPerNet) KD base image model DFF NetWarp TCB-ppm TCB-ocr SegFormer-b2 SegFormer-b3 ConvNeXt-S + UPerNet TFC
Builds on: ETC TFC Deep Feature Flow SegFormer
Closest papers by meaning
- Unleashing Multispectral Video's Potential in Semantic Segmentation: A Semi-supervised Viewpoint and New UAV-View Benchmark (NeurIPS 2024 · similarity 0.87)
- Exploiting Temporal State Space Sharing for Video Semantic Segmentation (CVPR 2025 · similarity 0.85)
- PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation (CVPR 2026 · similarity 0.85)
- Semantic and Sequential Alignment for Referring Video Object Segmentation (CVPR 2025 · similarity 0.85)
- Pay Attention to Target: Relation-Aware Temporal Consistency for Domain Adaptive Video Semantic Segmentation (AAAI 2024 · similarity 0.85)
- Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.