Research map / Detection and segmentation / Open-set and few-shot detection
Object Concepts Emerge from Motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns object-centric visual features from raw driving videos by turning optical-flow motion clusters into pseudo instance masks and training Swin/ResNet encoders with a contrastive pixel-embedding loss, then evaluating on depth, 3D detection and occupancy.
Problem
Existing visual foundation models (DINO, CLIP, MAE) capture semantic categories rather than object instances; the paper asks whether motion cues can supervise instance-level (object-centric) visual representations without labels or camera calibration.
Approach
Derive pseudo instance masks from VideoFlow optical flow via forward-backward consistency and BFS flow-similarity clustering, then train an encoder with a margin-based pixel contrastive loss so pixels of the same motion cluster embed together, with background handled by ignoring background-background pairs.
Key results
- On KITTI Eigen split with DCDepth, Ours Swin-L reaches SILog 6.558 and RMSE 1.929, beating Semantic-SAM Swin-L (6.713) and ImageNet-22K Swin-L (6.891).
- On nuScenes val with BEVFormerV2, Ours Swin-L achieves NDS 55.80 and mAP 47.29, versus ImageNet-22K Swin-L at 54.59 NDS and 45.91 mAP.
- On nuScenes occupancy with SparseOcc, Ours Swin-L attains RayIoU 38.7, close to DINOv2 ViT-L at 39.0 and above SimMIM Swin-L at 38.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use motion-derived object-centric pretraining instead of ImageNet-22K pretraining | BEVFormerV2 with Swin-T · nuScenes val | NDS: 51.69 -> 52.41 | Helped |
| Use motion-derived object-centric pretraining instead of ImageNet-22K pretraining | BEVFormerV2 with Swin-L · nuScenes val | NDS: 54.59 -> 55.80 | Helped |
| Use motion-derived object-centric pretraining instead of DINOv2 pretraining | BEVFormerV2 · nuScenes val, 704×256 | NDS: 51.91 -> 52.03 | Helped |
| Combine Ours features with DINO features (ablation) | DCDepth · KITTI Eigen split | SILog: 6.991 -> 6.796 | Helped |
| Combine Ours features with ImageNet-22K features (ablation) | DCDepth · KITTI Eigen split | SILog: 6.991 -> 6.927 | Helped |
| Combine DINO features with ImageNet-22K features (ablation) | DCDepth · KITTI Eigen split | SILog: 7.119 -> 7.071 | Mixed |
Surprising findings
- Although trained only on outdoor driving videos, features segment unseen indoor objects (windows, tools, cats, hands) and robot manipulation scenes.
- Combining the motion-based features with DINOv2 features in depth estimation yields the best SILog (6.796), suggesting the two representations are complementary.
Limitations
- Performance is limited by optical flow quality; extension to ego-centric and unconstrained web videos did not work well.
- Features emphasize whole objects, so part-level features are indistinguishable, making them unsuitable for precise keypoint matching.
- Pseudo-labels under-segment objects with weak motion or flow errors and only cover foreground instances with motion cues.
Experimental setup
Datasets: OpenDV-YouTube nuPlan KITTI nuScenes Ego4D RTX
Benchmarks: KITTI Eigen split KITTI online leaderboard nuScenes val 3D detection nuScenes val RayIoU
Metrics: SILog Abs Rel RMSE delta1 NDS mAP RayIoU
Baselines: ImageNet-22K supervised pretraining Semantic-SAM DINOv2 SimMIM MoCo v3 COCO supervised pretraining ImageNet-1K supervised pretraining
Builds on: VideoFlow DCDepth BEVFormer v2 SparseOcc SimMIM
Closest papers by meaning
- Object Pursuit: Building a Space of Objects via Discriminative Weight Generation (ICLR 2022 · similarity 0.88)
- Learning Features by Watching Objects Move (CVPR 2017 · similarity 0.88)
- Discovering Objects That Can Move (CVPR 2022 · similarity 0.88)
- Towards Stable Self-Supervised Object Representations in Unconstrained Egocentric Video (CVPR 2026 · similarity 0.87)
- Temporally Consistent Object-Centric Learning by Contrasting Slots (CVPR 2025 · similarity 0.86)
- Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.