Research map / Video / Video segmentation and tracking
FlowFeat: Pixel-Dense Embedding of Motion Profiles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills apparent-motion profiles from optical flow networks and unlabeled videos into a frozen-encoder DPT decoder, producing a 128-dim pixel-dense feature map (FlowFeat) that improves VOS, semantic segmentation and depth probing.
Problem
Vision encoders such as ViTs produce low-resolution feature grids that are suboptimal for dense prediction, and video-based representation learning is not yet dense, versatile and temporally consistent.
Approach
Train a lightweight DPT decoder on a frozen encoder so that, for each image, a per-iteration ridge-regression linear operator (computed with an EMA teacher on one crop) maps features to optical flow w.r.t. a random temporal neighbour; a focal gradient-matching loss sharpens motion boundaries. This embeds a distribution of plausible motions (motion profiles) without labels.
Key results
- On DAVIS-2017 linear probing, FlowFeat-K improves MAE-B16 JF by about 13.0 points and DINO2-S14 by a comparable margin to 66.1 JF versus 57.5 baseline.
- On NYUv2, FlowFeat-K reduces DINO-S16 RMSE from 0.4634 to 0.4422 (with ++ refinement) and DINO2-B14 RMSE from 0.3091 to 0.2791.
- On COCO-Stuff, FlowFeat-K raises DINO-S16 mIoU from 39.6 to 44.2 and FlowFeat-K++ further to 45.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FlowFeat-YT | DINO2-S14 · DAVIS-2017 val linear probing | JF: 57.5 -> 65.8 | Helped |
| add FlowFeat-K | DINO2-S14 · DAVIS-2017 val local KNN | JF: 65.1 -> 68.5 | Helped |
| add FlowFeat-YT | DINO-S16 · COCO-Stuff 2017 val | mIoU: 39.6 -> 44.7 | Helped |
| add FlowFeat-K | DINO2-B14 · NYUv2 val | RMSE: 0.3091 -> 0.2791 | Helped |
| add FlowFeat-K-based post-hoc refinement | DINO2-S14 segmentation probe · COCO-Stuff 2017 val | mIoU: 58.1 -> (59.6) | Helped |
| use a distribution of linear maps instead of a single map (ablation) | FlowFeat-YT · DAVIS-2017 val linear probing | JF: 56.7 -> 65.8 | Helped |
| use ridge regularization with gamma 1 instead of gamma 0.001 (ablation) | FlowFeat-YT · DAVIS-2017 val linear probing | JF: 58.2 -> 65.8 | Helped |
| add gradient matching loss (ablation) | FlowFeat-YT · DAVIS-2017 val linear probing | F_m: 67.7 -> 69.7 | Helped |
| use L1 instead of L2 reconstruction loss (ablation) | FlowFeat-YT · DAVIS-2017 val linear probing | JF: 63.3 -> 65.8 | Helped |
| add L1 reconstruction loss (ablation) | FlowFeat-YT · DAVIS-2017 val linear probing | JF: 65.3 -> 65.8 | Helped |
| replace SEA-RAFT with RAFT (ablation) | FlowFeat-YT · DAVIS-2017 val linear probing | JF: 65.8 -> 65.2 | Hurt |
| replace SEA-RAFT with unsupervised SMURF (ablation) | FlowFeat-YT · DAVIS-2017 val linear probing | JF: 65.8 -> 64.1 | Hurt |
Surprising findings
- Training with only the gradient loss (no L1 reconstruction term) causes only a small accuracy drop, suggesting the gradient term does most of the work.
- Neither the encoder nor FeatUp benefits from doubling the feature resolution in local KNN, whereas FlowFeat does.
Limitations
- Relies on brightness constancy and a pre-trained optical flow network, so may not transfer to medical, thermal or low-light domains.
- Frozen backbone caps accuracy, especially for high-frequency detail.
- Motion bias: emphasises large-motion foreground objects over static background.
Experimental setup
Datasets: YouTube-VOS Kinetics-400 DAVIS-2017 COCO-Stuff 2017 NYUv2
Benchmarks: DAVIS-2017 COCO-Stuff 2017 NYUv2
Metrics: J&F mean J mean F mean mIoU pixel accuracy RMSE delta threshold accuracy
Baselines: FeatUp LoftUp V-JEPA VideoMAE DINO DINOv2 MAE bilateral upsampling
Builds on: FeatUp SEA-RAFT DPT mean teacher PAMR
Closest papers by meaning
- Learning Motion Patterns in Videos (CVPR 2017 · similarity 0.88)
- Semantic Flow: Learning Semantic Fields of Dynamic Scenes from Monocular Videos (ICLR 2024 · similarity 0.87)
- Recurrent Partial Kernel Network for Efficient Optical Flow Estimation (AAAI 2024 · similarity 0.86)
- Flow-NeRF: Joint Learning of Geometry, Poses, and Dense Flow within Unified Neural Representations (CVPR 2025 · similarity 0.86)
- E-MoFlow: Learning Egomotion and Optical Flow from Event Data via Implicit Regularization (NeurIPS 2025 · similarity 0.86)
- DeltaFlow: An Efficient Multi-frame Scene Flow Estimation Method (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.