Research map / Video / Video segmentation and tracking
Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Opt-CWM learns image-conditioned Gaussian counterfactual perturbations, jointly with a flow-conditioned next-frame predictor, to extract self-supervised optical flow and occlusion from a pre-trained RGB next-frame predictor without labels or heuristics.
Problem
Self-supervised estimation of optical flow and occlusion from video without synthetic labels or hand-crafted heuristics such as photometric loss and smoothness regularization.
Approach
Parameterize the counterfactual perturbation in Counterfactual World Modeling as an input-conditioned Gaussian produced by an MLP on encoder tokens, and learn it end-to-end by passing the putative flow into a sparse flow-conditioned next-frame predictor whose RGB reconstruction loss supervises the perturbation generator, using a softargmax to keep the flow differentiable.
Key results
- On TAP-Vid DAVIS First, Opt-CWM reaches AJ 47.53 and AD 8.73, outperforming GMRW (36.47 AJ) and SMURF (30.64 AJ) and supervised SEA-RAFT (43.41 AJ).
- On TAP-Vid Kinetics First, Opt-CWM reaches AJ 44.85 versus 36.99 for SMURF among unsupervised methods.
- Replacing learned perturbations with fixed red or green squares drops DAVIS First AJ from 47.53 to 21.37 and 30.44, showing the value of learned counterfactuals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learned, image-conditioned perturbations instead of a red square (ablation) | Opt-CWM · TAP-Vid DAVIS First | AJ: 21.37 -> 40.51 | Helped |
| Use learned, image-conditioned perturbations instead of a green square (ablation) | Opt-CWM · TAP-Vid DAVIS First | AJ: 30.44 -> 40.51 | Helped |
| Increase multi-mask inference from 1 to 10 (ablation) | Opt-CWM · TAP-Vid DAVIS First | AJ: 42.85 -> 47.53 | Helped |
| Add four multiscale refinement iterations instead of none (ablation) | Opt-CWM · TAP-Vid DAVIS First | AJ: 32.71 -> 47.53 | Helped |
| Increase input resolution from 256 to 512 (ablation) | Opt-CWM · TAP-Vid DAVIS First | AJ: 37.00 -> 40.51 | Helped |
| Use temporally factored random masking instead of Video-MAE-style symmetric or tube masking (ablation) | CWM · TAP-Vid DAVIS First | AJ: 15.23 -> 37.00 | Helped |
| Set the inference masking ratio to 60% rather than the 90% reference (ablation) | Opt-CWM · TAP-Vid DAVIS First | AJ: 40.51 -> 43.28 | Helped |
| Distill Opt-CWM motion pseudo-labels into SEA-RAFT (ablation) | Opt-CWM · TAP-Vid First DAVIS | AJ: 30.64 -> 44.05 | Helped |
| Use Opt-CWM rather than SEA-RAFT | TAP-Vid First DAVIS · TAP-Vid First DAVIS | AJ: 43.41 -> 47.53 | Helped |
| Use Opt-CWM rather than SEA-RAFT | TAP-Vid First Kinetics · TAP-Vid First Kinetics | AJ: 39.27 -> 44.85 | Helped |
| Use Opt-CWM rather than SEA-RAFT | TAP-Vid First Kubric · TAP-Vid First Kubric | AJ: 75.64 -> 67.61 | Hurt |
| Use Opt-CWM rather than SEA-RAFT | TAP-Vid CFG DAVIS · TAP-Vid CFG DAVIS | AJ: 69.89 -> 69.53 | Hurt |
Surprising findings
- Fixed-perturbation CWM performs very poorly on TAP-Vid First (AJ 15.00 on DAVIS), far below the learned-perturbation version.
- Distilling Opt-CWM into a SEA-RAFT architecture with sparse pseudo-labels outperforms SMURF and is competitive with supervised RAFT on the CFG protocol.
Limitations
- Hand-designed fixed perturbations are out of distribution, motivating learned ones, but inference cost of multi-mask and multiscale refinement is high.
- Opt-CWM only outperforms supervised flow methods in some scenarios; on the CFG protocol it is competitive rather than uniformly best.
- Current work is restricted to two-frame base predictors and point-level flow.
Experimental setup
Datasets: Kinetics-400 TAP-Vid DAVIS TAP-Vid Kinetics TAP-Vid Kubric
Benchmarks: TAP-Vid DAVIS TAP-Vid Kinetics TAP-Vid Kubric
Metrics: AJ AD <delta_avg OA OF1
Baselines: RAFT SEA-RAFT SMURF Doduo GMRW CWM
Builds on: Counterfactual World Modeling VideoMAE SMURF GMRW SEA-RAFT
Closest papers by meaning
- Taming generative video models for zero-shot optical flow extraction (NeurIPS 2025 · similarity 0.88)
- Video Diffusion Models are Training-free Motion Interpreter and Controller (NeurIPS 2024 · similarity 0.85)
- TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.85)
- Moving Off-the-Grid: Scene-Grounded Video Representations (NeurIPS 2024 · similarity 0.85)
- No More Shortcuts: Realizing the Potential of Temporal Self-Supervision (AAAI 2024 · similarity 0.84)
- Visual Representation Learning with Stochastic Frame Prediction (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.