Research map / Detection and segmentation / Open-set and few-shot detection
Depth-aware Test-Time Training for Zero-shot Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DATTT, a two-stage zero-shot video object segmentation method that jointly learns mask and depth prediction with a depth modulation layer, then performs test-time training by enforcing depth consistency across augmentations, with momentum-based initialization and looping-through-video updates.
Problem
Zero-shot video object segmentation models trained on large video datasets generalize poorly to unseen videos; test-time training for ZSVOS lacks effective self-supervised objectives that suit video.
Approach
Uses depth-consistency across augmented views of the same frame as the self-supervised test-time training signal, with a depth-aware modulation layer that lets depth features modulate mask decoder features, plus video-specific TTT schedules (momentum-based weight initialization and loop-through-video).
Key results
- TTT-LTV on MiT-b1 improves Long-Videos J from 63.9 to 73.1 and FBMS from 75.1 to 76.9 in the ablation table (with depth and modulation).
- With Swin-T, DATTT reaches Long-Videos J 75.9 and SegTrackV2 gains +12.0 J over the baseline in TTT-LTV ablation.
- Compared with prior TTT methods (TENT, BN, TTT-Rot, TTT-MAE), DATTT gives consistent gains, while the others often drop performance after TTT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add depth prediction supervision to the two-stream model (ablation) | DATTT with Mit-b1 · DAVIS-16, FBMS, and Long-Videos | J: 75.9 -> 77.0; 75.1 -> 77.5; 63.9 -> 62.8 | Mixed |
| add the depth modulation layer (ablation) | DATTT with Mit-b1 · DAVIS-16, FBMS, and Long-Videos | J: 77.0 -> 77.2; 77.5 -> 78.0; 62.8 -> 70.5 | Mixed |
| use depth-aware test-time training (ablation) | DATTT with Mit-b1 · DAVIS-16, FBMS, and Long-Videos | J: 77.1 -> 77.5; 73.7 -> 76.9; 65.2 -> 73.1 | Mixed |
| use momentum-based weight initialization instead of per-frame initialization (ablation) | DATTT with Mit-b1 · FBMS, Long-Videos, MCL, SegTrackV2 | J: +2.3; +7.2; +7.6; +2.0 | Helped |
| loop through the video during test-time training (ablation) | DATTT with Mit-b1 · FBMS, Long-Videos, MCL, SegTrackV2 | J: +3.2; +7.9; +8.4; +4.4 | Helped |
| use TTT-N (ablation) | DATTT with Mit-b1 · DAVIS-16, FBMS, Long-Videos, MCL, SegTrackV2 | J: +0.3; +0.1; +1.3; +1.9; +1.0 | Helped |
| use TTT-LTV (ablation) | DATTT with Swin-T · DAVIS-16, FBMS, Long-Videos, MCL, SegTrackV2 | J: +0.2; +2.6; +12.4; +12.0; +1.5 | Helped |
| use TTT-MAE test-time training | Swin-T ZSVOS model · DAVIS-16, FBMS, Long-Videos, MCL, SegTrackV2 | J: −0.1; −0.3; −0.8; −0.9; −0.6 | Mixed |
| use TTT-Rot test-time training | Mit-b1 ZSVOS model · DAVIS-16, FBMS, Long-Videos, MCL, SegTrackV2 | J: −0.4; +0.4; +2.6; +4.0; −2.4 | Mixed |
| use depth-based DATTT rather than no TTT | Mit-b1 ZSVOS model · FBMS, Long-Videos, MCL, SegTrackV2 | J: +3.2; +7.9; +8.4; +4.4 | Helped |
| use depth-based DATTT rather than no TTT | Swin-T ZSVOS model · DAVIS-16, FBMS, Long-Videos, MCL, SegTrackV2 | J: +0.2; +2.6; +12.4; +12.0; +1.5 | Helped |
| use LiteMono instead of MonoDepth2 depth supervision (ablation) | DATTT · DAVIS-16, FBMS, Long-Videos | J: 76.8; 79.0; 68.1 | Mixed |
Surprising findings
- Prior image TTT methods (TENT, BN, TTT-Rot, TTT-MAE) yield little or negative gains on ZSVOS, which the authors attribute to minor domain shift in clean target datasets.
Limitations
- Depth supervision comes from monocular depth predictors and is noisy.
- Gains on DAVIS-16 are modest (about +0.2 to +0.4 J), and the method does not beat all ZSVOS state-of-the-art methods on every dataset.
Experimental setup
Datasets: YouTube-VOS DAVIS-16 FBMS Long-Videos MCL SegTrackV2
Benchmarks: DAVIS-16 FBMS Long-Videos MCL SegTrackV2
Metrics: J (region similarity) F (boundary accuracy)
Baselines: TENT BN TTT-Rot TTT-MAE 3DCSEG AGNN MATNet HFAN HCPN MED-VT
Builds on: TTT-Rot TTT-MAE SegFormer RAFT
Closest papers by meaning
- The Emergence of Objectness: Learning Zero-shot Segmentation from Videos (NeurIPS 2021 · similarity 0.86)
- Test-time Training for Matching-based Video Object Segmentation (NeurIPS 2023 · similarity 0.85)
- Object-Centric Refinement for Enhanced Zero-Shot Segmentation (ICLR 2026 · similarity 0.84)
- Learning Video Object Segmentation From Static Images (CVPR 2017 · similarity 0.83)
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos (CVPR 2025 · similarity 0.82)
- SMITE: Segment Me In TimE (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.