Research map / Video / Video generation
MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MeDM, which uses optical-flow-based Flow Coding and a closed-form averaging harmonization to enforce temporal consistency when running pre-trained image (Stable Diffusion + ControlNet) diffusion models frame by frame for zero-shot video-to-video translation.
Problem
Frame-wise image diffusion models produce flickering when used for video-to-video translation, and existing zero-shot video methods are either flicker-prone or complex; fine-tuning on video data is costly.
Approach
Encode video pixels into indices of a global pixel repository via per-frame optical flows and occlusions (Flow Coding), then at each denoising step replace the model's predicted samples/scores with the unweighted average over frames sharing the same repository entry (closed-form least-squares harmonization), with a latent-space workaround via autoencoder casting for LDMs.
Key results
- On MPI Sintel video rendering, the method achieves EPE 1.501 (estimated flow) and 1.456 (GT flow), versus 7.953 for Rerender A Video and 12.757 for ControlNet.
- In a human study with 71 participants on Sintel rendering, the method scored 4.423 video quality and 4.014 realism, above all baselines (best baseline Control-A-Video 3.662 quality).
- In video anonymization, recognizability dropped to 20.83% versus 63.01% for DeepPrivacy, with realism 3.507 vs 2.019.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MeDM temporal correspondence guidance with estimated optical flow | Stable Diffusion video rendering · MPI Sintel Rendering | EPE: 12.757 -> 1.501 | Helped |
| apply MeDM temporal correspondence guidance with estimated optical flow | Stable Diffusion video rendering · Virtual KITTI 2 Rendering | EPE: 12.575 -> 2.857 | Helped |
| add MeDM temporal correspondence guidance | Stable Diffusion assistive rendering · MPI Sintel Assistive Rendering | EPE: 5.924 -> 1.570 | Helped |
| add MeDM temporal correspondence guidance | Stable Diffusion assistive rendering · Virtual KITTI 2 Assistive Rendering | EPE: 5.070 -> 2.695 | Helped |
| use GT rather than estimated optical flow | MeDM · MPI Sintel Rendering | EPE: 1.501 -> 1.456 | Helped |
| add InstructPix2Pix conditioning instead of Lineart conditioning | MeDM video editing · DAVIS 2016 text-guided video editing | text alignment: 3.810 -> 4.338 | Helped |
| add SDEdit (ablation) | MeDM with ControlNet · MPI Sintel Assistive Rendering | EPE: 1.456 -> 1.202 | Helped |
| add ControlNet (ablation) | MeDM with SDEdit · MPI Sintel Assistive Rendering | EPE: 1.798 -> 1.202 | Helped |
| remove both SDEdit and ControlNet (ablation) | MeDM · MPI Sintel rendering | EPE: 1.202 -> 3.781 | Hurt |
| increase temporal correspondence guidance strength w (ablation) | ControlNet video rendering · MPI Sintel | EPE: 12.757 -> 1.310 | Helped |
| use local rather than global harmonization (ablation) | MeDM · MPI Sintel | EPE: 1.456 -> 1.515 | Hurt |
| decrease local harmonization kernel ratio r (ablation) | MeDM local harmonization · MPI Sintel | EPE: 1.614 -> 1.515 | Helped |
Surprising findings
- Using extrapolation instead of interpolation for guidance led to serious early saturation, so the paper uses interpolation.
- Distant-frame optical flows did not necessarily improve results because they exhibit greater estimation errors.
Limitations
- Relies on discretized per-pixel optical flow, causing rounding/discretization errors in code propagation.
- Inherits biases of the underlying diffusion models; does not model shading changes from lighting.
- Optical flows may not align with objects undergoing significant contour changes, limiting structural translation.
Experimental setup
Datasets: MPI Sintel Virtual KITTI 2 DAVIS 2016 CelebV-HQ
Benchmarks: MPI Sintel Virtual KITTI 2 DAVIS 2016 CelebV-HQ
Metrics: end-point error (EPE) of flow estimated on generated videos human-rated video quality realism text alignment recognizability faithfulness
Baselines: ControlNet ControlVideo Video ControlNet Rerender A Video Control-A-Video Pix2Video DeepPrivacy SDEdit
Builds on: MultiDiffusion ControlNet SDEdit Stable Diffusion DDIM
Closest papers by meaning
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.91)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.90)
- LLM-grounded Video Diffusion Models (ICLR 2024 · similarity 0.90)
- Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search (NeurIPS 2025 · similarity 0.90)
- Hierarchical Patch Diffusion Models for High-Resolution Video Generation (CVPR 2024 · similarity 0.90)
- FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.