Research map / Video / Video generation
Frame In-N-Out: Unbounded Controllable Image-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Frame In-N-Out, an image-to-video task where objects exit or enter the frame under motion trajectory control, built on CogVideoX-I2V with an unbounded canvas, identity reference conditioning, and a newly curated dataset and benchmark.
Problem
Existing motion-controlled and elements-to-video image-to-video models are bounded by the first frame border and cannot make objects leave the frame (Frame Out) or enter it with a new identity reference (Frame In) under trajectory control.
Approach
Extends the conditioning canvas beyond the first frame (unbounded canvas), encodes pixel-aligned motion trajectories as channel-concatenated VAE latents, and injects identity references as extra frames in the 3D full-attention DiT, trained with a two-stage curriculum and a full-field loss over the canvas.
Key results
- On Frame Out, Stage 2 reaches FVD 318.38 and trajectory error 17.85 vs. best baseline ToRA 566.78 and 40.72 respectively.
- On Frame In, the method reaches FID 30.84, FVD 227.30, VLM 0.863, versus Phantom+motion prompt FID 72.84 and VLM 0.540.
- Human study win rates vs. baselines range from 73.3% (ToRA) to 100% (ImageConductor).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Stage 2 unbounded canvas and identity conditioning after Stage 1 motion training | Frame In-N-Out on CogVideoX-I2V · Frame Out | Trajectory Error: 48.46 -> 17.85 | Helped |
| Add Stage 2 unbounded canvas and identity conditioning after Stage 1 motion training | Frame In-N-Out on CogVideoX-I2V · Frame Out | FID: 38.36 -> 32.02 | Helped |
| Add motion description prompt | SkyReels-A2 · Frame In | FID: 74.00 -> 61.20 | Helped |
| Add motion description prompt | Phantom · Frame In | FID: 69.55 -> 72.84 | Hurt |
| Test at 384x480 inference resolution instead of the baseline 448x640 (ablation) | Frame In-N-Out · Frame In | VLM: 0.868 -> 0.797 | Hurt |
| Train without full field loss (ablation) | Frame In-N-Out · Frame In | Trajectory Error: 9.54 -> 48.49 | Hurt |
| Train without full field loss (ablation) | Frame In-N-Out · Frame In | VSeg MAE: 0.0107 -> 0.0497 | Hurt |
| Train without full field loss (ablation) | Frame In-N-Out · Frame In | VLM: 0.868 -> 0.792 | Hurt |
| Create new absolute position encoding instead of interpolating the learned encoding (ablation) | Frame In-N-Out · Frame In | FID: 30.18 -> 36.03 | Hurt |
| Create new absolute position encoding instead of interpolating the learned encoding (ablation) | Frame In-N-Out · Frame In | Relative DINO: 0.68 -> 0.97 | Hurt |
Surprising findings
- Rel. DINO for the proposed method is slightly worse than Phantom with motion prompts (1.62 vs 1.39), attributed to identity moving outside the canvas.
- A naive first-frame-only loss failed to give stable results, so the full-canvas loss was adopted.
Limitations
- Single trajectory point causes 3D pose ambiguity (back vs. side view) and size ambiguity for the identity reference.
- Some examples show color distortion outside the first frame region in the full-canvas generation.
Experimental setup
Datasets: OpenVid-1M VidGen-1M WebVid-10M COCO
Benchmarks: Frame In-N-Out evaluation set (183 Frame In, 189 Frame Out cases, from OpenVid-1M reserved subset)
Metrics: FID FVD LPIPS Trajectory Error VSeg MAE VLM accuracy Relative DINO human win rate
Baselines: DragAnything Image Conductor ToRA Phantom SkyReels-A2
Builds on: CogVideoX Concat-ID Marigold
Closest papers by meaning
- Follow-Your-Click: Open-domain Regional Image Animation via Motion Prompts (AAAI 2025 · similarity 0.88)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.88)
- FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis (CVPR 2025 · similarity 0.88)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.88)
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.88)
- Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.