Research map / 3D vision / Depth and stereo
ProMotion: Prototypes As Motion Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProMotion, a unified transformer framework that models optical flow and scene depth via a feature denoiser and a prototypical learner, with transfer to 2D/3D downstream detection and segmentation.
Problem
Motion learning (optical flow and scene depth) is handled by task-specific architectures, and pixel-wise feature matching is hurt by photometric inconsistency and uncertainty; a unified motion representation is lacking.
Approach
Model motion as a set of semantic prototypes obtained from a denoised, orthogonally projected token subspace (feature denoiser via subspace self-attention plus ISTA-style large-margin projection), clustered with Sinkhorn-Knopp, then decoded by a shared adaptation head for both flow and depth.
Key results
- On Sintel test with C+T+S+K(+H) training, ProMotion reports AEPE 1.04 (clean) and 2.01 (final), versus 1.20 and 2.12 for FlowFormer.
- On KITTI-15 test flow, ProMotion achieves Fl-all 4.30, versus 4.68 for FlowFormer.
- Sintel depth Abs Rel 0.54 vs 0.62 for DPT-Hybrid; KITTI depth Abs Rel 0.054 vs 0.058 for AdaBins.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Feature Denoiser (ablation) | ProMotion baseline · Sintel val flow clean | AEPE: 0.54 -> 0.51 | Helped |
| add Prototypical Learner after Feature Denoiser (ablation) | ProMotion · Sintel val flow clean | AEPE: 0.51 -> 0.45 | Helped |
| add unified learning to the component setup (ablation) | ProMotion · Sintel val flow clean | AEPE: 0.45 -> 0.40 | Helped |
| replace subspace self-attention with vanilla self-attention (ablation) | ProMotion · Sintel val flow clean | AEPE: 0.40 -> 0.45 | Hurt |
| replace subspace self-attention with multi-head self-attention (ablation) | ProMotion · Sintel val flow clean | AEPE: 0.40 -> 0.42 | Hurt |
| replace subspace self-attention with spatial reduction attention (ablation) | ProMotion · Sintel val flow clean | AEPE: 0.40 -> 0.43 | Hurt |
| increase prototype count from 10 to 100 (ablation) | ProMotion · Sintel val flow clean | AEPE: 0.46 to 0.40 | Helped |
| increase prototype count from 100 to 200 (ablation) | ProMotion · Sintel val flow clean | AEPE: 0.40 -> 0.41 | Hurt |
| increase decoder iterations from 1 to 12 (ablation) | ProMotion · Sintel val flow clean | AEPE: 0.58 to 0.40 | Helped |
| increase decoder iterations from 12 to 24 (ablation) | ProMotion · Sintel val flow clean | AEPE: 0.40 -> 0.39 | Helped |
| replace Sinkhorn-Knopp with cosine similarity prototype updating (ablation) | ProMotion · Sintel val flow clean | AEPE: 0.40 -> 0.43 | Hurt |
| replace Sinkhorn-Knopp with K-Means prototype updating (ablation) | ProMotion · Sintel val flow clean | AEPE: 0.40 -> 0.41 | Hurt |
Limitations
- Iterative decoding with more steps stretches training and inference time.
- Depth gains on KITTI are modest relative to saturated baselines.
Experimental setup
Datasets: FlyingChairs FlyingThings3D Sintel KITTI HD1K Eigen split
Benchmarks: Sintel KITTI-15 Sintel depth KITTI depth
Metrics: AEPE Fl-all Fl-epe Abs Rel Sq Rel RMSE
Baselines: RAFT FlowFormer MatchFlow GMFlow AdaBins DPT-Hybrid P3Depth TransDepth
Builds on: RAFT FlowFormer DPT AdaBins Tweedie's formula
Closest papers by meaning
- Prototypical Transformer As Unified Motion Learners (ICML 2024 · similarity 0.92)
- Video Diffusion Models are Training-free Motion Interpreter and Controller (NeurIPS 2024 · similarity 0.87)
- Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.87)
- Motion Diversification Networks (CVPR 2024 · similarity 0.86)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.86)
- DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.