Research map / Video / Video generation
Dimension-Reduction Attack! Video Generative Models are Experts on Controllable Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Repurposes the HunyuanVideo-I2V video generator for controllable image synthesis by treating condition and target images as boundary frames of a mixup-interpolated shot transition, with frame-skip position embeddings and attention masking.
Problem
Video generative models hold rich spatiotemporal priors, but adapting them to controllable image generation is hindered by the mismatch between continuous video frames and the abrupt, discrete change between condition and target images.
Approach
Treat condition and target images as endpoints of a synthetic fade-in-fade-out shot transition built by gamma-corrected mixup with smoothstep weights, skip temporal RoPE intervals (FSPE) to reduce transition frames, reweight the loss by target content proportion, and mask cross-attention between condition/target image and prompt tokens.
Key results
- On DreamBench, DRA-Ctrl achieves VL score 2.44 (vs. 2.34 for the I2V baseline and 2.01 for the T2V baseline) with CLIP-T 0.298.
- On COCO2017 deblurring, DRA-Ctrl reports MSE 11 and FID 9.08 versus OminiControl's 132 and 11.49.
- Generation with 4 latent frames (delta=12) is reported as 90.4% faster than the 37-latent-frame I2V variant (24.0 s vs 251 s) while attaining the highest VL score.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Repurpose the video model with DRA-Ctrl for Canny conditioning | HunyuanVideo-I2V · COCO2017 validation, Canny | F1: 0.38 -> 0.42 | Helped |
| Repurpose the video model with DRA-Ctrl for depth conditioning | HunyuanVideo-I2V · COCO2017 validation, Depth | MSE: 903 -> 76 | Helped |
| Repurpose the video model with DRA-Ctrl for deblurring | HunyuanVideo-I2V · COCO2017 validation, Deblur | MSE: 132 -> 11 | Helped |
| Repurpose the video model with DRA-Ctrl for colorization | HunyuanVideo-I2V · COCO2017 validation, Colorization | MSE: 24 -> 30 | Hurt |
| Repurpose the video model with DRA-Ctrl for mask-conditioned generation | HunyuanVideo-I2V · COCO2017 validation, Mask | MSE: 6248 -> 16 | Helped |
| Use fade-in-fade-out rather than slide-away transitions (ablation) | DRA-Ctrl · DreamBench subject-driven generation | VL Score: 2.19 -> 2.42 | Helped |
| Use 8 rather than 4 transition frames (ablation) | DRA-Ctrl · DreamBench subject-driven generation | VL Score: 2.19 -> 2.42 | Helped |
| Use 12 rather than 8 transition frames (ablation) | DRA-Ctrl · DreamBench subject-driven generation | VL Score: 2.42 -> 2.09 | Hurt |
| Add loss reweighting (ablation) | DRA-Ctrl · DreamBench subject-driven generation | VL Score: 2.32 -> 2.42 | Helped |
| Add FSPE (ablation) | DRA-Ctrl · DreamBench subject-driven generation | VL Score: 2.28 -> 2.42 | Helped |
| Add the mixup strategy (ablation) | DRA-Ctrl · DreamBench subject-driven generation | VL Score: 2.38 -> 2.42 | Helped |
| Add attention masking (ablation) | DRA-Ctrl · DreamBench subject-driven generation | VL Score: 2.41 -> 2.42 | Helped |
Surprising findings
- Fade-in-fade-out mixup transitions outperform slide-away transitions, which the authors attribute to video models having stronger priors for fade transitions.
- The I2V baseline's DINO score (0.803) exceeds the Oracle real-image DINO score (0.774), reflecting over-copying of the condition image.
Limitations
- Slightly inferior FID and SSIM compared with image-specific methods.
- CLIP-T prompt adherence is marginally lower than competing methods, attributed to LLaVA-based prompt understanding.
- Transition frames reduce generation efficiency relative to image models.
Experimental setup
Datasets: Text-to-Image-2M Subjects200K COCO2017 validation DreamBench
Benchmarks: COCO2017 validation DreamBench
Metrics: F1 MSE FID SSIM DINO CLIP-I CLIP-T VL score
Baselines: ControlNet T2I-Adapter Uni-ControlNet OminiControl EasyControl FramePainter ObjectMover T2V baseline I2V baseline
Builds on: HunyuanVideo-I2V flow matching 3D RoPE LoRA mixup
Closest papers by meaning
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.87)
- Language Model Beats Diffusion - Tokenizer is key to visual generation (ICLR 2024 · similarity 0.86)
- Plug-and-Play Context Feature Reuse for Efficient Masked Generation (NeurIPS 2025 · similarity 0.86)
- Grid Diffusion Models for Text-to-Video Generation (CVPR 2024 · similarity 0.86)
- VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis (ICLR 2025 · similarity 0.85)
- Training-Free Efficient Video Generation via Dynamic Token Carving (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.