Research map / Video / Video segmentation and tracking
Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Temporal Feature Correlation (TFC), which uses cosine similarity of low-level features between keyframe and non-keyframe to linearly interpolate reused features and sparsely update low-similarity points with multi-scale deformable attention, improving efficiency and stability in video semantic segmentation on VSPW.
Problem
Efficient video semantic segmentation with the keyframe paradigm suffers from feature misalignment between reused and recomputed features, and feature reuse yields brittle prediction consistency, creating a stability-performance dilemma.
Approach
Use cosine similarity between keyframe and current low-level features as a label-consistency signal: linearly interpolate high-level decoder features weighted by that similarity (no learnable parameters), and apply sparse multi-scale deformable attention only to low-similarity points, supervised by a focal-loss BCE on the similarity.
Key results
- On VSPW with MiT-B0, SegFormer + TFC improves mVC8 from 82.7 to 85.3 and mVC16 from 77.3 to 80.0 while keeping mIoU at 32.9 and raising FPS from 37.5 to 38.2.
- With MiT-B5, TFC raises mVC8 from 87.8 to 90.5 and mVC16 from 83.7 to 86.7 at mIoU 48.3 (vs 48.2) and FPS 37.8 vs 26.2 for the image-based SegFormer.
- Feature reuse alone lowers mIoU to 31.5 on MiT-B0, while TFC recovers accuracy to 32.9 with higher consistency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TFC to SegFormer | SegFormer with MiT-B0 · VSPW validation | mVC8: 82.7 -> 85.3 (+2.6) | Helped |
| add TFC to SegFormer | SegFormer with MiT-B1 · VSPW validation | mVC8: 84.7 -> 86.8 (+2.1) | Helped |
| add TFC to SegFormer | SegFormer with MiT-B5 · VSPW validation | mVC16: 83.7 -> 86.7 (+3.0) | Helped |
| add cosine-similarity-weighted linear interpolation (ablation) | Feature Reuse, MiT-B0 · VSPW validation | mVC8: 84.3 -> 85.0 | Helped |
| replace weighted interpolation with hard thresholding (ablation) | Feature Reuse, MiT-B0 · VSPW validation | mVC8: 84.3 -> 83.7 | Hurt |
| replace similarity-based weights with fixed interpolation weights (ablation) | Feature Reuse, MiT-B0 · VSPW validation | mIoU: 31.5 -> 31.0 | Hurt |
| add sparse MSDA with threshold 0.8 (ablation) | Feature Reuse, MiT-B0 · VSPW validation | mIoU: 31.5 -> 32.7 | Helped |
| use sparse MSDA threshold 1.0 (ablation) | Feature Reuse, MiT-B0 · VSPW validation | mVC8: 84.3 -> 79.0 | Hurt |
| replace sparse MSDA with flow-guided warping (ablation) | Feature Reuse, MiT-B0 · VSPW validation | mVC8: 84.3 -> 81.0 | Hurt |
| compute cosine similarity from x1 instead of x2 (ablation) | TFC, MiT-B0 · VSPW validation | mIoU: 32.9 -> 31.6 | Hurt |
| remove focal-loss-based binary cross-entropy supervision (ablation) | TFC, MiT-B0 · VSPW validation | mVC8: 85.3 -> 83.1 | Hurt |
| increase keyframe interval from 4 to 9 (ablation) | TFC, MiT-B0 · VSPW validation | mVC8: 85.3 -> 86.6 | Helped |
Surprising findings
- Using all points for sparse MSDA (threshold 1.0) increases mIoU to 33.8 but drops mVC8 to 79.0, showing a trade-off between accuracy and stability.
- Non-keyframe performance can exceed keyframe performance when interpolation is unrestricted, which breaks the feature reuse assumption.
Limitations
- Errors propagate from keyframes to non-keyframes, which the authors leave for future work.
- Evaluated only on VSPW, the only densely annotated VSS dataset available at the time.
Experimental setup
Datasets: VSPW ImageNet
Benchmarks: VSPW
Metrics: mIoU wIoU mVC8 mVC16 FPS Params GFLOPs
Baselines: SegFormer Feature Reuse TDNet MRCFA CFFM
Builds on: SegFormer Deformable DETR (multi-scale deformable attention) Clockwork convnets (feature reuse) TDNet Focal loss
Closest papers by meaning
- Simultaneously Short- and Long-Term Temporal Modeling for Semi-Supervised Video Semantic Segmentation (CVPR 2023 · similarity 0.88)
- Feature Space Optimization for Semantic Video Segmentation (CVPR 2016 · similarity 0.88)
- Pay Attention to Target: Relation-Aware Temporal Consistency for Domain Adaptive Video Semantic Segmentation (AAAI 2024 · similarity 0.88)
- Learning Spatial-Semantic Features for Robust Video Object Segmentation (ICLR 2025 · similarity 0.87)
- Semantic and Sequential Alignment for Referring Video Object Segmentation (CVPR 2025 · similarity 0.87)
- Exploiting Temporal State Space Sharing for Video Semantic Segmentation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.