Research map / Video / Video segmentation and tracking
VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free video semantic segmentation using pre-trained SD 2.1 or SVD diffusion features with a scene context KNN model, correspondence-based refinement, and masked modulation.
Problem
Frame-by-frame diffusion-based image segmentation yields temporally inconsistent predictions on videos, and no training-free diffusion-based video semantic segmentation method existed.
Approach
Builds an autoregressively updated KNN scene context model on aggregated diffusion features, refines coarse per-frame masks via correspondence-based temporal voting, and upsamples with masked latent blending in the modulation process, all without training.
Key results
- Ours (SD) reaches 60.6 mIoU on VSPW versus 43.4 for EmerDiff (SD) among training-free methods, and is comparable to supervised UniVS (59.8) and near DVIS++ (63.8).
- Ours (SD) achieves 37.3 mIoU on Cityscapes and 92.3 mVC8 on CamVid, outperforming EmerDiff (SD) at 21.5 mIoU on Cityscapes.
- Ablation shows masked modulation, feature aggregation, and CBR together give the best mVC consistency (SD mVC8 89.8 vs 76.9 baseline at batch size 14).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VidSeg rather than EmerDiff with the same SD 2.1 backbone | SD 2.1 video segmentation · VSPW val | mIoU: 43.4 -> 60.6 | Helped |
| Use VidSeg rather than EmerDiff with the same SVD backbone | SVD video segmentation · VSPW val | mIoU: 39.7 -> 53.2 | Helped |
| Use VidSeg rather than EmerDiff with the same SD 2.1 backbone | SD 2.1 video segmentation · Cityscapes val | mIoU: 21.5 -> 37.3 | Helped |
| Use VidSeg rather than EmerDiff with the same SVD backbone | SVD video segmentation · Cityscapes val | mIoU: 11.0 -> 36.2 | Helped |
| Use VidSeg rather than EmerDiff with the same SD 2.1 backbone | SD 2.1 video segmentation · CamVid val | mIoU: 6.9 -> 20.6 | Helped |
| Use VidSeg rather than EmerDiff with the same SVD backbone | SVD video segmentation · CamVid val | mIoU: 7.3 -> 16.6 | Helped |
| Use SD 2.1 rather than SVD as the backbone | VidSeg · VSPW val | mIoU: 53.2 -> 60.6 | Helped |
| Add masked modulation to VidSeg (ablation) | VidSeg with batch size 14, without feature aggregation or CBR · VSPW first 30 val videos | mIoU: 43.4 -> 45.6 | Helped |
| Add feature aggregation to VidSeg (ablation) | VidSeg with batch size 14, without masked modulation or CBR · VSPW first 30 val videos | mIoU: 43.4 -> 45.2 | Helped |
| Add CBR to VidSeg (ablation) | VidSeg with batch size 14, without masked modulation or feature aggregation · VSPW first 30 val videos | mIoU: 43.4 -> 44.6 | Helped |
| Increase batch size from 1 to 14 (ablation) | VidSeg without masked modulation, feature aggregation, or CBR · VSPW first 30 val videos | mIoU: 33.4 -> 43.4 | Helped |
| Add masked modulation (ablation) | VidSeg with SVD, batch size 14, without feature aggregation or CBR · VSPW first 30 val videos | mIoU: 38.9 -> 38.6 | No clear effect |
Surprising findings
- SVD temporal-layer features are unstable across frames, so the spatial features were used instead.
- Feature aggregation slightly lowered mIoU on VSPW due to coarse ground-truth annotations, though it helped mVC in combination.
Limitations
- Depends on the quality of image inversion and VAE compression, which discards fine details.
- Instance-agnostic: groups all objects of the same class into one cluster.
- Struggles with small objects at the latent compression factor, and with challenging lighting in driving datasets.
Experimental setup
Datasets: VSPW Cityscapes CamVid
Benchmarks: VSPW Cityscapes CamVid
Metrics: mIoU mVC8 mVC16
Baselines: EmerDiff CLIPpy TMANet UniVS DVIS++
Builds on: EmerDiff Stable Diffusion Stable Video Diffusion CLIPpy
Closest papers by meaning
- Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.89)
- EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models (ICLR 2024 · similarity 0.89)
- Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.89)
- InvSeg: Test-Time Prompt Inversion for Semantic Segmentation (AAAI 2025 · similarity 0.88)
- Exploiting Temporal State Space Sharing for Video Semantic Segmentation (CVPR 2025 · similarity 0.88)
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.