Research map / Video / Video segmentation and tracking
Pay Attention to Target: Relation-Aware Temporal Consistency for Domain Adaptive Video Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PAT, a domain-adaptive video semantic segmentation method combining a target domain focalizer (target-aware prototypes via attention) with relation-aware temporal consistency based on class-ranking permutation distributions, reaching 53.0 mIoU on VIPER→Cityscapes-Seq and 54.9 on SYNTHIA-Seq→Cityscapes-Seq.
Problem
Domain adaptive video semantic segmentation suffers from unreliable pseudo-labels and weak temporal supervision signals (hard pseudo-labels or per-class KL) when adapting from synthetic to real videos without target annotations.
Approach
Generate target-aware prototypes via two-stage cross-attention pooling to make pseudo-labels more accessible, and enforce temporal consistency on class-ranking permutation distributions (inter-class relations) rather than per-class scores.
Key results
- On VIPER→Cityscapes-Seq, PAT reaches 53.0 mIoU vs 51.1 for SFC (prior SOTA in table) and 37.4 source-only.
- On SYNTHIA-Seq→Cityscapes-Seq, PAT reaches 54.9 mIoU, 1.1 above TPS (53.8).
- Ablation on VIPER: target domain focalizer adds about 2.5 mIoU over baseline (48.9→51.4) and relation-aware temporal consistency adds 4.1 mIoU (48.9→53.0 with both components on baseline path).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply PAT instead of I2VDA | DAVSS on VIPER → Cityscapes-Seq · VIPER → Cityscapes-Seq | mIoU: +1.8% mIoU | Helped |
| apply PAT instead of TPS | DAVSS on SYNTHIA-Seq → Cityscapes-Seq · SYNTHIA-Seq → Cityscapes-Seq | mIoU: 1.1% mIoU | Helped |
| add TDF to the baseline (ablation) | PAT on VIPER → Cityscapes-Seq · VIPER → Cityscapes-Seq | mIoU: 48.9 -> 51.4 | Helped |
| add RTC to the baseline (ablation) | PAT on VIPER → Cityscapes-Seq · VIPER → Cityscapes-Seq | mIoU: 48.9 -> 51.7 | Helped |
| use both TDF and RTC rather than TDF alone (ablation) | PAT on VIPER → Cityscapes-Seq · VIPER → Cityscapes-Seq | mIoU: 51.4 -> 53.0 | Helped |
| increase serial TDFs from 1 to 2 (ablation) | PAT on VIPER → Cityscapes-Seq · VIPER → Cityscapes-Seq | mIoU: 53.0 -> 53.2 | No clear effect |
| increase serial TDFs from 1 to 3 (ablation) | PAT on VIPER → Cityscapes-Seq · VIPER → Cityscapes-Seq | FPS: 1.04 -> 0.87 | Hurt |
| replace hard pseudo-labeling with soft pseudo-label temporal consistency (L2 or KL) (ablation) | PAT on VIPER → Cityscapes-Seq · VIPER → Cityscapes-Seq | mIoU: 51.3, 51.6, 51.9 | Helped |
| use RTC instead of hard pseudo-labeling (ablation) | PAT on VIPER → Cityscapes-Seq · VIPER → Cityscapes-Seq | mIoU: 51.3 -> 53.0 | Helped |
| set TDF threshold τ to 0 rather than 0.8 (ablation) | PAT on VIPER → Cityscapes-Seq · VIPER → Cityscapes-Seq | mIoU: 53.0 -> 51.7 | Hurt |
| vary the TDF threshold τ from 0.6 to 0.9 (ablation) | PAT on VIPER → Cityscapes-Seq · VIPER → Cityscapes-Seq | mIoU: 52.6, 52.7, 53.0, 52.9 | No clear effect |
| vary trade-off weight λ from 0.5 to 1.5 (ablation) | PAT on VIPER → Cityscapes-Seq · VIPER → Cityscapes-Seq | mIoU: 52.5, 53.0, 52.7 | No clear effect |
Limitations
- Full permutation computation over all K classes is computationally prohibitive, so only top-4 classes are used.
- Additional TDF stages give negligible gains while reducing training speed.
Experimental setup
Datasets: VIPER SYNTHIA-Seq Cityscapes-Seq
Benchmarks: VIPER→Cityscapes-Seq SYNTHIA-Seq→Cityscapes-Seq
Metrics: mIoU
Baselines: Src.-only AdvEnt FDA RDA PixMatch DA-VSN I2VDA TPS SFC
Builds on: Accel PixMatch TPS SFC ADVENT
Closest papers by meaning
- Self-Supervised Augmentation Consistency for Adapting Semantic Segmentation (CVPR 2021 · similarity 0.89)
- Learning to Adapt via Latent Domains for Adaptive Semantic Segmentation (NeurIPS 2021 · similarity 0.89)
- On the Road to Online Adaptation for Semantic Image Segmentation (CVPR 2022 · similarity 0.89)
- Pixel-Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation (NeurIPS 2020 · similarity 0.89)
- Exploit Domain-Robust Optical Flow in Domain Adaptive Video Semantic Segmentation (AAAI 2023 · similarity 0.89)
- Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.