Research map / Video / Video segmentation and tracking
Prototypical Transformer As Unified Motion Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProtoFormer, a Transformer that replaces self-attention with EM-style cross-attention prototyping plus latent synchronization, targeting optical flow and depth with extensions to tracking and video stabilization.
Problem
Pixel-wise feature matching in motion tasks (optical flow, depth) is corrupted by photometric and geometric inconsistencies such as shadows and occlusions, and existing methods are task-specific rather than unified.
Approach
Reformulate Transformer attention as EM clustering: learnable prototypes are iteratively updated by cross-attention to image features (E-step soft assignment, M-step prototype update), and features are then refined by masked cross-attention to prototypes (latent synchronization), built on the Twins backbone.
Key results
- On Sintel (C+T+S+K+H) ProtoFormer reports 0.48 clean and 0.69 final train EPE, versus CRAFT at 0.60 and 1.06 in the same setting.
- On KITTI Eigen depth, ProtoFormer reports Abs Rel 0.062 and RMSE 2.716, compared with P3Depth at 0.071 and 2.842.
- Ablations show cross-attention prototyping and latent synchronization each improve over the Base model, with the full model reaching 0.48/0.69 on Sintel clean/final.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Cross-Attention Prototyping to Base (ablation) | ProtoFormer · Sintel flow ablation | Clean EPE: 0.55 -> 0.51 | Helped |
| Add Latent Synchronization to Base (ablation) | ProtoFormer · Sintel flow ablation | Final EPE: 0.81 -> 0.77 | Helped |
| Replace Cross-Attention Prototyping with cosine similarity updating (ablation) | ProtoFormer · Sintel flow ablation | Clean EPE: 0.48 -> 0.51 | Hurt |
| Replace Cross-Attention Prototyping with vanilla cross-attention (ablation) | ProtoFormer · Sintel flow ablation | Clean EPE: 0.48 -> 0.50 | Hurt |
| Replace Cross-Attention Prototyping with Criss Cross-Attention (ablation) | ProtoFormer · Sintel flow ablation | Clean EPE: 0.48 -> 0.50 | Hurt |
| Replace Cross-Attention Prototyping with K-Means (ablation) | ProtoFormer · Sintel flow ablation | Clean EPE: 0.48 -> 0.49 | Hurt |
| Increase prototype iterations from 1 to 4 (ablation) | ProtoFormer · Sintel flow ablation | Clean EPE: 0.52 -> 0.48 | Helped |
| Increase prototype count from 10 to 100 (ablation) | ProtoFormer · Sintel flow ablation | Clean EPE: 0.53 -> 0.48 | Helped |
| Remove Latent Synchronization (ablation) | ProtoFormer · Sintel flow ablation | Final EPE: 0.69 -> 0.74 | Hurt |
| Replace Latent Synchronization with a vanilla FC layer (ablation) | ProtoFormer · Sintel flow ablation | Final EPE: 0.69 -> 0.73 | Hurt |
| Add Cross-Attention Prototyping to Base (ablation) | ProtoFormer · KITTI depth ablation | Abs Rel: 0.074 -> 0.067 | Helped |
| Add Latent Synchronization to Base (ablation) | ProtoFormer · KITTI depth ablation | Abs Rel: 0.074 -> 0.071 | Helped |
Surprising findings
- On Sintel depth, ProtoFormer's Abs Rel improves but its RMSE (0.486) is worse than AdaBins-level baselines only in some columns; the paper reports the best results on most but not all metrics.
Limitations
- Gains on some metrics are small, and the model does not beat all baselines on every column (e.g., KITTI-15 F1-all vs. some baselines).
- Object tracking and video stabilization results are only in the appendix and are described as competitive rather than best.
- Discriminative design may struggle with out-of-distribution or open-set inputs, per the authors.
Experimental setup
Datasets: FlyingChairs FlyingThings Sintel KITTI-2015 HD1K VKITTI KITTI Eigen split LaSOT TrackingNet DeepStab
Benchmarks: Sintel KITTI-2015 KITTI Eigen depth LaSOT TrackingNet DeepStab
Metrics: EPE F1-all Abs Rel RMSE delta1 Success Precision Distortion Value Stability Score
Baselines: RAFT FlowFormer CRAFT MatchFlow SKFlow GMFlow AdaBins P3Depth Perceiver IO UTT StabNet PWStableNet
Builds on: Twins RAFT Perceiver IO AdaBins
Closest papers by meaning
- ProMotion: Prototypes As Motion Learners (CVPR 2024 · similarity 0.92)
- DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation (NeurIPS 2026 · similarity 0.84)
- Understanding Video Transformers via Universal Concept Discovery (CVPR 2024 · similarity 0.84)
- PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos (CVPR 2024 · similarity 0.84)
- DyST: Towards Dynamic Neural Scene Representations on Real-World Videos (ICLR 2024 · similarity 0.83)
- MotionDreamer: One-to-Many Motion Synthesis with Localized Generative Masked Transformer (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.