Research map / Video / Video segmentation and tracking
Infer from What You Have Seen Before: Temporally-dependent Classifier for Semi-supervised Video Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a temporally-dependent classifier (TDC) whose context prototypes come from the previous frame's predictions, implicitly enforcing temporal consistency for semi-supervised video semantic segmentation.
Problem
Semi-supervised semantic segmentation of video with very few pixel-level labels; existing methods treat video as independent frames in the classifier and ignore temporal relations at the classification stage.
Approach
Replace the per-frame conventional classifier with a temporally-dependent classifier that combines learned global prototypes with context prototypes computed on-the-fly from the previous frame's pseudo labels, plus a bidirectional (forward and backward) variant and joint-prototype label relaxation, trained inside a FixMatch-style pseudo-label framework.
Key results
- On Cityscapes with 1/30 labels, the method reaches 55.81% mIoU vs 45.73% for the Accel supervised baseline (+10.08) and 52.86% for IFR.
- On CamVid with 1/30 labels, the method reaches 50.95% mIoU vs 42.37% baseline and 49.50% for IFR.
- Ablation on Cityscapes 1/30 with PSPNet-ResNet18 shows components raise mIoU from 43.68 to 51.29 when combining global prototypes, context prototypes and bidirectional constraint.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TDC to Accel | Accel · Cityscapes val, 1/30 labeled | mIoU: 52.86 -> 55.81 | Helped |
| Add TDC to Accel | Accel · CamVid test, 1/8 labeled | mIoU: 57.37 -> 59.88 | Helped |
| Add context prototypes to global prototypes (ablation) | TDC · Cityscapes val, 1/30 labeled | mIoU: 46.78 (+3.10) -> 50.14 (+6.46) | Helped |
| Add bidirectional constraint (ablation) | TDC with global and context prototypes · Cityscapes val, 1/30 labeled | mIoU: 50.14 (+6.46) -> 51.29 (+7.61) | Helped |
| Increase unsupervised-loss weight from λ=0.0 to λ=0.2 (ablation) | TDC · Cityscapes val, 1/30 labeled | mIoU: 43.68 -> 51.29 | Helped |
| Increase unsupervised-loss weight from λ=0.2 to λ=0.5 or λ=1.0 (ablation) | TDC · Cityscapes val, 1/30 labeled | mIoU: 51.29 -> 50.74; 48.54 | Hurt |
| Increase sampled clip length from 1 to 5 frames (ablation) | TDC · Cityscapes val, 1/30 labeled | mIoU: 46.87 -> 51.29 | Helped |
| Increase sampled clip length from 5 to 7 or 10 frames (ablation) | TDC · Cityscapes val, 1/30 labeled | mIoU: 51.29 -> 50.93; 50.64 | Hurt |
| Apply TDC to video segmentation architectures (ablation) | DFF, Accel, and DAVSS · Cityscapes val, 1/30 labeled | mIoU: 44.19 -> 52.63; 45.73 -> 55.81; 46.51 -> 55.25 | Helped |
| Add TDC to Accel | Accel · Cityscapes validation subset | temporal consistency: 70.04 -> 75.38 | Helped |
Surprising findings
- Using only global prototypes within TDC already acts like a pseudo-label scheme, and the context prototypes add a large further gain, suggesting the classifier design itself drives the improvement.
- Increasing the number of unlabeled frames per clip beyond 5 degrades performance due to training instability.
Limitations
- Gains are evaluated only on Cityscapes and CamVid with a single video segmentation architecture family (Accel-based), with ablations on PSPNet-ResNet18.
- Inference uses the conventional classifier; TDC is only used during training, so the benefit is representational rather than an inference-time mechanism.
Experimental setup
Datasets: Cityscapes CamVid
Benchmarks: Cityscapes CamVid
Metrics: mIoU temporal consistency (TC) score
Baselines: Accel (supervised) CCT CAC CPS IFR DFF DAVSS
Builds on: IFR Accel FixMatch CPS
Closest papers by meaning
- Simultaneously Short- and Long-Term Temporal Modeling for Semi-Supervised Video Semantic Segmentation (CVPR 2023 · similarity 0.88)
- Weakly Supervised Semantic Segmentation Using Web-Crawled Videos (CVPR 2017 · similarity 0.87)
- Semi-Supervised Video Semantic Segmentation With Inter-Frame Feature Reconstruction (CVPR 2022 · similarity 0.87)
- A Transductive Approach for Video Object Segmentation (CVPR 2020 · similarity 0.86)
- Boosting Semi-Supervised Semantic Segmentation with Probabilistic Representations (AAAI 2023 · similarity 0.85)
- Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.