Research map / Video / Video generation
VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoSwap, a diffusion-based framework for customized video subject swapping that uses sparse semantic point correspondences (with user point removal/dragging) instead of dense correspondences, enabling shape-changing swaps.
Problem
Diffusion-based video editing relies on dense correspondences that enforce source shape, making subject swapping with shape change (and customized identities) ineffective.
Approach
Uses a small set of semantic points (DIFT-embedded, tracked with CoTracker) as sparse motion guidance injected into the encoder of a motion-layer-augmented image diffusion model via learnable MLPs, with user interactions (point removal, dragging propagated through a layered neural atlas) to handle partial or shape-changing correspondences.
Key results
- Human evaluation vs. Tune-A-Video: VideoSwap preferred 78% vs 22% overall; vs. Rerender-A-Video 84% vs 16%.
- Against DDIM+T2I-Adapter on AnimateDiff, overall preference 66% vs 34%; against DDIM+Tune-A-Video 69% vs 31%.
- Semantic point registration with learned MLPs reaches similar results to randomly initialized learnable embeddings with 3x less registration time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoSwap instead of Tune-A-Video | Customized video subject swapping · Video subject swapping | Overall preference: 78% v.s. 22% | Helped |
| Use VideoSwap instead of FateZero | Customized video subject swapping · Video subject swapping | Overall preference: 70% v.s. 30% | Helped |
| Use VideoSwap instead of Text2Video-Zero | Customized video subject swapping · Video subject swapping | Overall preference: 80% v.s. 20% | Helped |
| Use VideoSwap instead of Rerender-A-Video | Customized video subject swapping · Video subject swapping | Overall preference: 84% v.s. 16% | Helped |
| Use sparse motion features rather than encoding point maps with T2I-Adapter (ablation) | VideoSwap · Video subject swapping | Video quality | Helped |
| Use DIFT point embeddings rather than randomly initialized learnable embeddings (ablation) | VideoSwap · Point registration | Registration time: 3× less time | Helped |
| Remove point patch loss (ablation) | VideoSwap · Video subject swapping | Source-structure leakage | Hurt |
| Register points only on early sampling steps using the semantic-enhanced schedule (ablation) | VideoSwap · Video subject swapping | Point alignment | Helped |
| Drag semantic points and propagate their displacement over time (ablation) | VideoSwap · Black swan-to-duck swap | Target identity | Helped |
| Use the unmodified source point sequence instead of dragging points (ablation) | VideoSwap · Black swan-to-duck swap | Target identity | Hurt |
Limitations
- Dense/structure-preserving priors are abandoned, so swaps may leak or misalign if semantic point correspondence is partial or inaccurate
- Requires user-defined keyframe semantic points and optional manual dragging, which adds intervention
Experimental setup
Datasets: real-world source videos (unnamed)
Benchmarks: human evaluation on customized video subject swapping (1000 MTurk questionnaires) automatic evaluation (in supplementary)
Metrics: subject identity motion alignment temporal consistency overall preference
Baselines: Tune-A-Video FateZero Text2Video-Zero Rerender-A-Video TokenFlow StableVideo DDIM+Tune-A-Video DDIM+T2I-Adapter
Builds on: AnimateDiff ED-LoRA DIFT CoTracker Layered Neural Atlas Tune-A-Video T2I-Adapter
Closest papers by meaning
- VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models (CVPR 2025 · similarity 0.87)
- MotionEditor: Editing Video Motion via Content-Aware Diffusion (CVPR 2024 · similarity 0.87)
- HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness (NeurIPS 2024 · similarity 0.87)
- ConMo: Controllable Motion Disentanglement and Recomposition for Zero-Shot Motion Transfer (CVPR 2025 · similarity 0.85)
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.85)
- Multi-subject Open-set Personalization in Video Generation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.