Research map / Video / Video segmentation and tracking
Zero-Shot Scene Change Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free zero-shot scene change detection method that repurposes the DEVA tracking model (with SAM masks) by feeding reference and query images as a pair, plus an adaptive content threshold and style bridging layers, and extends it to video.
Problem
Scene change detection normally requires training on labeled SCD datasets, which is costly and fails to generalize across style variations (season, weather, lighting) between image pairs.
Approach
Treat SCD as tracking: feed reference and query images to a tracking model in place of consecutive frames, so missing/new objects are found by mask non-survival; add a sequence-length-dependent content threshold for abrupt content changes and training-free AdaIN-based style bridging layers to handle style gaps.
Key results
- On ChangeSim in-domain, the training-free method reaches 35.8 mIoU on Normal and 31.6 on Dusty-air, beating C-3PO (32.7 and 29.7), but trails on Low-illumination (25.2 vs 27.1).
- Cross-domain, C-3PO drops to around 24-27 mIoU when trained on a different subset, while the zero-shot method keeps 35.8/31.6/25.2 across subsets.
- On VL-CMU-CD and PCD, the method averages 54.0 F1 versus 53.4 for C-3PO, which only does well in-domain (79.4 and 82.4 F1) and drops to 24.3 and 11.6 cross-domain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use zero-shot tracking-based SCD instead of in-domain C-3PO | ChangeSim Normal · ChangeSim Normal (in-domain) | mIoU: 32.7 -> 35.8 | Helped |
| Use zero-shot tracking-based SCD instead of in-domain C-3PO | ChangeSim Dusty-air · ChangeSim Dusty-air (in-domain) | mIoU: 29.7 -> 31.6 | Helped |
| Use zero-shot tracking-based SCD instead of in-domain C-3PO | ChangeSim Low-illumination · ChangeSim Low-illumination (in-domain) | mIoU: 27.1 -> 25.2 | Hurt |
| Train C-3PO on the test domain rather than a different ChangeSim domain | C-3PO · ChangeSim cross-domain | mIoU | Helped |
| Add both ACT and SBL (ablation) | Zero-shot SCD on ChangeSim · ChangeSim average | Average mIoU: 25.9 -> 30.9 | Helped |
| Add SBL (ablation) | Zero-shot SCD on ChangeSim · ChangeSim Dusty-air and Low-illumination | mIoU: 24.4 -> 27.0; 25.0 -> 25.7 | Helped |
| Add SBL (ablation) | Zero-shot SCD on ChangeSim · ChangeSim Normal | mIoU: 28.2 -> 27.1 | Hurt |
| Add ACT (ablation) | Zero-shot SCD on ChangeSim · ChangeSim Normal | mIoU: 28.2 -> 36.0 | Helped |
| Add ACT (ablation) | Zero-shot SCD on ChangeSim · ChangeSim Low-illumination | mIoU: 25.0 -> 21.6 | Hurt |
| Use SBL in all four encoder locations rather than no SBL (ablation) | Zero-shot SCD on ChangeSim · ChangeSim average | Average mIoU: 25.3 -> 30.9 | Helped |
| Extend image-based SCD to video with T_max 60 (ablation) | Zero-shot SCD · ChangeSim average | Average mIoU: 28.2 -> 30.9 | Helped |
| Increase T_max beyond 60 (ablation) | Zero-shot video SCD · ChangeSim average | Average mIoU: 30.9; 30.8; 29.8 | Mixed |
Surprising findings
- The adaptive content threshold hurts performance on the low-illumination subset where tracking is poor, decreasing that subset's score from 25.0 to 21.6 when used alone versus the baseline without ACT and SBL.
- Increasing the video sequence length beyond 60 frames gives little benefit or slightly degrades performance.
Limitations
- Performance is lower than the trained baseline on the low-illumination subset because tracking quality degrades there.
- Relies on the quality of SAM mask proposals and the tracker; small masks and mask overlap needed post-processing.
- Video sequence length is capped at T_max=60 to limit memory, with camera motion breaking the comparison.
Experimental setup
Datasets: ChangeSim VL-CMU-CD PCD
Benchmarks: ChangeSim VL-CMU-CD PCD
Metrics: mIoU F1
Baselines: C-3PO
Builds on: DEVA Segment Anything Model AdaIN C-3PO
Closest papers by meaning
- Segment Any Change (NeurIPS 2024 · similarity 0.89)
- Towards Generalizable Scene Change Detection (CVPR 2025 · similarity 0.88)
- Change3D: Revisiting Change Detection and Captioning from A Video Modeling Perspective (CVPR 2025 · similarity 0.88)
- Multi-View Pose-Agnostic Change Localization with Zero Labels (CVPR 2025 · similarity 0.84)
- Adaptive Visual Scene Understanding: Incremental Scene Graph Generation (NeurIPS 2024 · similarity 0.84)
- Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.