Research map / Video / Video segmentation and tracking
What If: Understanding Motion Through Sparse Interactions
Stefan Andreas Baumann, Nick Stracke, Timy Phan, Björn Ommer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize and jointly train the vision encoder from pre-trained DINOv2 instead of from scratch (ablation) | FPT · TalkingHead-1KH, 5 pokes (250k steps) | EPE: 6.14 -> 5.38 | Improved |
| unfreeze the pre-trained vision encoder (joint training) instead of freezing it (ablation) | FPT · TalkingHead-1KH, 1 poke (250k steps) | EPE: 8.30 -> 8.08 | Improved |
| unfreeze the pre-trained vision encoder instead of freezing it (ablation) | FPT · TalkingHead-1KH, 10 pokes (250k steps) | EPE: 3.78 -> 3.99 | Hurt |
| use 4 GMM components instead of 1 (ablation) | FPT · TalkingHead-1KH, 1 poke (250k steps) | EPE: 7.60 -> 8.08 | Hurt |
| use 4 GMM components instead of 1 (ablation) | FPT · TalkingHead-1KH, 100 pokes (250k steps) | EPE: 2.59 -> 2.33 | Improved |
| increase GMM components from 4 to 16 (ablation) | FPT · TalkingHead-1KH, 1 poke (250k steps) | EPE: 8.08 -> 8.41 | Hurt |
| use 2 GMM components instead of 4 (ablation) | FPT · TalkingHead-1KH, 1 poke (250k steps) | EPE: 8.08 -> 8.98 | Hurt |
| use full covariance instead of diagonal covariance (ablation) | FPT · TalkingHead-1KH, 100 pokes (250k steps) | EPE: 2.24 -> 2.33 | Hurt |
| use full covariance instead of diagonal covariance (ablation) | FPT · TalkingHead-1KH, 1 poke (250k steps) | EPE: 8.13 -> 8.08 | No clear effect |
| use KL-divergence score instead of flow-magnitude thresholding for moving part segmentation (ablation) | FPT · Drag-A-Move moving part segmentation | mIoU: 0.415 -> 0.572 | Improved |
| halve or double the segmentation threshold (ablation) | FPT · Drag-A-Move moving part segmentation | mIoU: 0.54/0.52 | No clear effect |
| fine-tune FPT on Drag-A-Move for 30k steps (ablation) | FPT · Drag-A-Move motion estimation | EPE: 12.74 -> 3.57 | Improved |
Closest papers by meaning
- iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis (ICCV 2021 · similarity 0.88)
- The Right Spin: Learning Object Motion from Rotation-Compensated Flow Fields (IJCV 2023 · similarity 0.86)
- Understanding Object Dynamics for Interactive Image-to-Video Synthesis (CVPR 2021 · similarity 0.86)
- ProMotion: Prototypes As Motion Learners (CVPR 2024 · similarity 0.86)
- Motion Modes: What Could Happen Next? (CVPR 2025 · similarity 0.86)
- PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Model (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
