Research map / Video / Video segmentation and tracking
Implicit Motion Function
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Implicit Motion Function (IMF), an encoder-decoder video model that represents inter-frame motion as low-dimensional latent tokens and models frame correlation via cross-attention in the decoder, applied to reconstruction, compression, and talking-head editing.
Problem
Optical-flow-based video modeling is inefficient, lacks sparsity, cannot model the probabilistic intrinsic motion of objects or new content, and encoder-centric frameworks give limited generative capability and redundant intermediate representations.
Approach
Replace optical flow and keypoints with a learned latent token per frame (encoded independently), and perform frame-to-frame correlation implicitly in the decoder via a latent token decoder (style-modulated convolutions) plus cross-attention alignment of reference appearance features to the current frame.
Key results
- On VFHQ reconstruction, IMF reaches PSNR 26.86 and LPIPS 0.0540 versus 23.46 PSNR and 0.0823 LPIPS for the best prior baseline (IW).
- On general datasets, IMF achieves the lowest L1 and LPIPS on Flower, Wavecloth and Foliage compared with TPSMM, IW, and others.
- In video compression on VFHQ, IMF achieves a better rate-distortion trade-off than VTM and DCVC-DC in the reported curves.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use IMF instead of IW | video reconstruction · CelebV-HQ | LPIPS: 0.1287 -> 0.1057 | Helped |
| use IMF instead of PECHead | video reconstruction · VFHQ | LPIPS: 0.0847 -> 0.0540 | Helped |
| use IMF instead of IW | video reconstruction · Flower | LPIPS: 0.1983 -> 0.1746 | Helped |
| use IMF instead of IW | video reconstruction · Wavecloth | LPIPS: 0.1380 -> 0.1265 | Helped |
| use IMF instead of IW | video reconstruction · Foliage | LPIPS: 0.1479 -> 0.1209 | Helped |
| add implicit motion representation to explicit representation and alignment (ablation) | IMF on Flower · Flower | L1: 0.0642 -> 0.0544 | Helped |
| add implicit motion alignment to implicit motion representation (ablation) | IMF on Flower · Flower | LPIPS: 0.1921 -> 0.1746 | Helped |
| add implicit motion representation to explicit motion alignment (ablation) | IMF on Flower · Flower | LPIPS: 0.1762 -> 0.1746 | Helped |
| add implicit motion representation and implicit motion alignment (ablation) | IMF on VFHQ · VFHQ | L1: 0.0435 -> 0.0283 | Helped |
| add implicit motion alignment to implicit motion representation (ablation) | IMF on VFHQ · VFHQ | LPIPS: 0.0612 -> 0.0540 | Helped |
| add implicit motion representation to explicit motion alignment (ablation) | IMF on VFHQ · VFHQ | LPIPS: 0.0713 -> 0.0540 | Helped |
Surprising findings
- Explicit keypoint or optical flow methods fail to model hands and new content such as appearing objects, which the implicit latent token approach captures.
Limitations
- Talking head editing evaluated qualitatively only on FFHQ images, with no quantitative editing metrics reported.
- Ablation results show trade-offs in some metrics (e.g., explicit-representation variants sometimes score better on individual metrics).
Experimental setup
Datasets: CelebV-HQ VFHQ Flower Wavecloth Foliage FFHQ
Benchmarks: CelebV-HQ VFHQ Flower Wavecloth Foliage FFHQ
Metrics: L1 SSIM PSNR FID LPIPS BPP
Baselines: FOMM MRAA Face-Vid2Vid LIA IW PECHead TPSMM VTM DCVC-DC
Builds on: FOMM StyleGAN2 Implicit Warping (IW) DCVC-DC Transformer
Closest papers by meaning
- Generalizable Implicit Motion Modeling for Video Frame Interpolation (NeurIPS 2024 · similarity 0.87)
- Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think (CVPR 2025 · similarity 0.86)
- Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation (CVPR 2025 · similarity 0.86)
- Bias for Action: Video Implicit Neural Representations with Bias Modulation (CVPR 2025 · similarity 0.86)
- Disentangled Motion Modeling for Video Frame Interpolation (AAAI 2025 · similarity 0.85)
- MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.