Research map / Video / Video generation
FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FRESCO, a zero-shot video-to-video translation framework on Stable Diffusion that adds intra-frame spatial correspondence to inter-frame optical-flow correspondence via attention guidance and gradient-based feature optimization.
Problem
Zero-shot text-guided video translation with image diffusion models suffers from temporal inconsistency, since prior methods only constrain inter-frame correspondence via optical flow, which is inconsistent, undercovers occlusions, and accumulates errors.
Approach
Combine intra-frame spatial correspondence (self-similarity of the input frame) with inter-frame optical-flow correspondence, enforced both by FRESCO-guided attention (spatial-guided, efficient cross-frame, temporal-guided) and by explicit gradient-descent optimization of U-Net decoder features, plus batched multi-frame processing and keyframe-based long video translation.
Key results
- FRESCO attains Fram-Acc 0.978, Tem-Con 0.975, Pixel-MSE 0.012 and Spat-Con 0.0805, best on all metrics except none, vs Rerender-A-Video (0.955, 0.969, 0.016, 0.0836).
- In a 57-participant user study, FRESCO was preferred in 65.0% of comparisons vs 23.3% for Rerender-A-Video.
- Ablation Pixel-MSE on hair strands drops from 0.037 (cross-frame attention only) to 0.012 (full FRESCO).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add FRESCO spatial-temporal correspondence guidance | Zero-shot video translation · 23 videos | Fram-Acc: 0.955 -> 0.978 | Helped |
| Add FRESCO spatial-temporal correspondence guidance | Zero-shot video translation · 23 videos | Tem-Con: 0.969 -> 0.975 | Helped |
| Add temporal-related adaptation (ablation) | FRESCO ablation baseline · Video translation ablation | Pixel-MSE: 0.032 -> 0.015 | Helped |
| Add spatial-related adaptation (ablation) | FRESCO ablation baseline · Video translation ablation | Pixel-MSE: 0.032 -> 0.020 | Helped |
| Add attention adaptation (ablation) | FRESCO ablation baseline · Video translation ablation | Pixel-MSE: 0.032 -> 0.016 | Helped |
| Add feature optimization (ablation) | FRESCO ablation baseline · Video translation ablation | Pixel-MSE: 0.032 -> 0.019 | Helped |
| Combine all spatial-temporal adaptations (ablation) | FRESCO ablation baseline · Video translation ablation | Pixel-MSE: 0.032 -> 0.012 | Helped |
| Increase temporal-guided attention scale λt (ablation) | FRESCO · Two transformed frames | Pixel-MSE: 0.016, 0.014, 0.013, 0.012 | Helped |
| Increase spatial-guided attention scale λs (ablation) | FRESCO · Cat translation | Pose consistency | Helped |
| Use efficient cross-frame attention instead of previous-frame-only attention (ablation) | FRESCO · Newly appearing fingers | Attended region size: 41.6% | Helped |
| Optimize features before attention layers rather than after attention layers (ablation) | FRESCO · Video translation | Feature optimization quality | Helped |
| Optimize decoder features rather than the compact predicted latent x̂′0 (ablation) | FRESCO · Video translation | Visual artifact quality | Helped |
Surprising findings
- Rerender-A-Video outperforms FRESCO when high-quality optical flow is available, since it aligns frames at the pixel level.
Limitations
- Underperforms Rerender-A-Video when optical flow is high quality.
- Does not support large shape deformations or significant appearance changes due to spatial correspondence constraint.
- Relies on optical flow estimation (GMFlow), which is unreliable under occlusion or rapid motion.
Experimental setup
Metrics: Fram-Acc Tem-Con Pixel-MSE Spat-Con user preference
Baselines: Text2Video-Zero ControlVideo Rerender-A-Video
Builds on: Stable Diffusion ControlNet SDEdit Rerender-A-Video FLATTEN Text2Video-Zero GMFlow
Closest papers by meaning
- MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance (AAAI 2024 · similarity 0.89)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.88)
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.88)
- Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer (CVPR 2024 · similarity 0.88)
- One-Step Diffusion for Detail-Rich and Temporally Consistent Video Super-Resolution (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.