Research map / Video / Video segmentation and tracking
SMITE: Segment Me In TimE
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SMITE, which adapts an inflated text-to-image diffusion UNet (Stable Diffusion) with learned segment tokens and fine-tuned cross-attention to segment unseen videos at arbitrary granularity from one or few reference images, using point-tracking-based temporal voting and low-pass regularization at inference.
Problem
Video object part segmentation at arbitrary/flexible granularity, where masks are defined by only one or a few reference images not drawn from the target video, while keeping labels temporally consistent across frames.
Approach
Combine a few-shot diffusion segmentation method (SLiMe-style learned text tokens over WAS maps) with an inflated spatio-temporal UNet, two-phase training that also tunes cross-attention, and inference-time guidance that tracks CoTracker trajectories over attention maps with temporal voting plus a DCT low-pass regularizer.
Key results
- On SMITE-50 with 10 reference images, SMITE reaches 75.14 mean mIOU and 0.82 mean F-measure, versus 66.62 for SLiMe frame-by-frame (Baseline-I) and 66.53 for Grounded SAM 2.
- Ablation on SMITE-50 shows mIOU rising from 66.22 (SLiMe) to 69.36 (+inflated UNet), 71.70 (+cross-attention tuning), 72.64 (+tracking), and 75.14 (+low-pass regularization).
- In a 1-shot setting SMITE reaches 58.55 mIOU, and 5-shot reaches 73.88 mIOU on SMITE-50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SMITE instead of GSAM2 | Video segmentation on SMITE-50 · SMITE-50 Mean | mIOU: 66.53 -> 75.14 | Helped |
| Use SMITE instead of Baseline-III (SLiMe + Inflated UNet) | Video segmentation on SMITE-50 · SMITE-50 Mean | mIOU: 69.36 -> 75.14 | Helped |
| Add an Inflated UNet to Baseline-I (ablation) | SLiMe on SMITE-50 · SMITE-50 | mIOU: 66.22 -> 69.36 | Helped |
| Add cross-attention tuning (ablation) | SLiMe + Inflated UNet on SMITE-50 · SMITE-50 | mIOU: 69.36 -> 71.70 | Helped |
| Add tracking (ablation) | SLiMe + Inflated UNet + cross-attention tuning · SMITE-50 | mIOU: 71.70 -> 72.64 | Helped |
| Add low-pass regularization (ablation) | SMITE with tracking on SMITE-50 · SMITE-50 | mIOU: 72.64 -> 75.14 | Helped |
| Train with 1 reference image instead of 10 (ablation) | SMITE on SMITE-50 · SMITE-50 | mIOU: 75.14 -> 58.55 | Hurt |
| Train with 5 reference images instead of 10 (ablation) | SMITE on SMITE-50 · SMITE-50 | mIOU: 75.14 -> 73.88 | Hurt |
| Use joint training instead of two-phase training (ablation) | SMITE on SMITE-50 · SMITE-50 | mIOU: 75.14 -> 62.17 | Hurt |
Surprising findings
- Directly applying CoTracker in pixel space alongside SLiMe (Baseline-II) does not improve over SLiMe alone, which the authors attribute to pixel-color drift.
- On DAVIS-2016 zero-shot, SMITE trails specialized VOS methods (80.8 J vs. 83.4-86.6 for SOTA), and struggles with small objects and thin structures.
Limitations
- Tracking and voting are limited by the performance of the off-the-shelf tracker (CoTracker).
- Low-resolution cross-attention maps reduce accuracy for small or fine segments.
- Inference is slow and requires per-video diffusion and optimization, so it is not interactive.
Experimental setup
Datasets: SMITE-50 PASCAL-Part DAVIS-2016 PUMaVOS
Benchmarks: SMITE-50 PUMaVOS DAVIS-2016 PASCAL-Part
Metrics: mIOU F-measure J F user preference ranking
Baselines: SLiMe (Baseline-I) SLiMe + CoTracker (Baseline-II) Inflated UNet SLiMe (Baseline-III) Grounded SAM 2 XMem++ SegGPT OIParts ReGAN SegDDPM
Builds on: SLiMe CoTracker Stable Diffusion Grounded SAM 2
Closest papers by meaning
- Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.87)
- Segmenting Moving Objects via an Object-Centric Layered Representation (NeurIPS 2022 · similarity 0.87)
- Breaking the "Object" in Video Object Segmentation (CVPR 2023 · similarity 0.87)
- Track and Segment: An Iterative Unsupervised Approach for Video Object Proposals (CVPR 2016 · similarity 0.87)
- Using Diffusion Priors for Video Amodal Segmentation (CVPR 2025 · similarity 0.86)
- Segment Any Motion in Videos (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.