Research map / Video / Video segmentation and tracking
CoDeF: Content Deformation Fields for Temporally Consistent Video Processing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoDeF, a hash-based canonical content field plus 3D temporal deformation field that represents a video so that image algorithms (ControlNet, SAM, Real-ESRGAN) can be applied once to the canonical image and propagated to all frames for temporally consistent video processing.
Problem
Image algorithms do not transfer to video with high temporal consistency; existing video editing via generative models suffers from flicker, and layered neural atlases have limited reconstruction fidelity and produce unnatural canonical images.
Approach
Factorize a video into a 2D canonical image field and a 3D hash-encoded temporal deformation field, with annealed hash encoding and flow-guided consistency loss to keep the canonical image natural and the deformation smooth, so an image algorithm applied once to the canonical image propagates across the whole video.
Key results
- Video reconstruction PSNR on DAVIS is about 2.3 dB higher than Layered Neural Atlas; e.g., Boat 34.13 vs 31.51 dB.
- Text-guided translation frame consistency CLIP score 98.54 for CoDeF vs 99.17 for Text2Live, with user score for temporal consistency 4.76 (highest).
- Per-video optimization takes about 5 minutes on a single A6000 GPU versus about 10 hours for the layered neural atlas.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace positional encoding with 3D hash encoding (ablation) | CoDeF deformation field · Video reconstruction | PSNR: +3.1 dB | Helped |
| remove annealed hash encoding (ablation) | CoDeF · Canonical image | naturalness | Hurt |
| remove flow loss (ablation) | CoDeF · Smooth regions in video | temporal consistency | Hurt |
| increase grouped deformation fields from K=1 to K=2 (ablation) | CoDeF · Blackswan | PSNR: 31.51 -> 31.97 | Helped |
| increase grouped deformation fields from K=1 to K=2 (ablation) | CoDeF · Boat | PSNR: 34.13 -> 34.73 | Helped |
| increase grouped deformation fields from K=1 to K=2 (ablation) | CoDeF · Kite-surf | PSNR: 34.26 -> 34.35 | Helped |
| use CoDeF canonical-image lifting instead of Tune-A-Video | Video translation · Video translation | Frame Consistency User Score: 2.12 -> 4.76 | Helped |
| use CoDeF canonical-image lifting instead of Text2LIVE | Video translation · Video translation | Textual Alignment User Score: 2.32 -> 4.13 | Helped |
| use CoDeF canonical-image lifting instead of Text2LIVE | Video translation · Video translation | Textual Alignment CLIP Score: 30.72 -> 27.43 | Hurt |
Surprising findings
- Text2Live scores higher on frame-consistency CLIP score (99.17) than CoDeF (98.54), despite CoDeF winning the user study.
Limitations
- Blurred or incoherent canonical images on complex scenes with scale changes, new objects, or many small fast-moving entities.
- Per-scene optimization is required for each video.
- Struggles with extreme viewpoint changes and large non-rigid deformations.
Experimental setup
Datasets: DAVIS OVIS
Benchmarks: DAVIS
Metrics: PSNR CLIP score user score (temporal consistency and textual alignment)
Baselines: Text2Live Tune-A-Video FateZero Layered Neural Atlas ControlNet per-frame
Builds on: Neural Layered Atlas dynamic NeRF multi-resolution hash encoding RAFT ControlNet
Closest papers by meaning
- HAIF-GS: Hierarchical and Induced Flow-Guided Gaussian Splatting for Dynamic Scene (NeurIPS 2025 · similarity 0.85)
- HIMMEL: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding (NeurIPS 2026 · similarity 0.84)
- Implicit Motion Function (CVPR 2024 · similarity 0.84)
- Restage4D: Reanimating Deformable 3D Reconstruction from a Single Video (NeurIPS 2025 · similarity 0.83)
- TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene (NeurIPS 2024 · similarity 0.83)
- MorpheuS: Neural Dynamic 360deg Surface Reconstruction from Monocular RGB-D Video (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.