Research map / Video / Video generation
CAGE: Unsupervised Visual Composition and Animation for Controllable Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains CAGE, an unsupervised controllable video diffusion (flow-matching) transformer conditioned on sparse, randomly cropped DINOv2 spatial tokens, enabling scene composition and object animation without annotations.
Problem
Controllable video generation typically needs annotations (text, boxes, motion); existing unsupervised methods use motion-based controls and cannot compose scenes and animate objects with the same control signal.
Approach
Use sparse DINOv2 spatial tokens of object parts placed at desired space-time locations as a unified control format, computed on random crops to remove positional information, so the model inpaints missing content and learns compositionality and dynamics without any annotations.
Key results
- On CLEVRER, CAGE achieves FVD 62 and FID 4.3 versus YODA's 70 and 5.7 when generating 15 frames from one frame.
- On BAIR, CAGE with 10% controls reaches LPIPS 0.107, FID 6.4, FVD 136, outperforming YODA (FID 18.2, FVD 264) and other baselines.
- On EPIC-KITCHENS, CAGE improves over YODA with FID 95 vs 152 and FVD 393 vs 664.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAGE rather than YODA | video generation · CLEVRER | SSIM: 0.93 -> 0.98 | Helped |
| Use CAGE rather than YODA | video generation · CLEVRER | LPIPS: 0.126 -> 0.166 | Hurt |
| Use CAGE rather than YODA | video generation · CLEVRER | FVD: 70 -> 62 | Helped |
| Use CAGE rather than YODA | video generation · EPIC-KITCHENS | FID: 152 -> 95 | Helped |
| Reduce conditioning to 1% controls from 10% controls | CAGE · BAIR | FID: 6.4 -> 7.2 | Hurt |
| Use tracking controls rather than 10% future-frame controls | CAGE · BAIR | FVD: 136 -> 214 | Hurt |
| Add scale/position invariance through random crops (ablation) | CAGE · CLEVRER composition | CTRL: 1.306 -> 1.615 | Helped |
| Add scale/position invariance through random crops (ablation) | CAGE · CLEVRER composition | FID: 12.88 -> 4.98 | Helped |
| Use randomized t_i rather than shared noise levels (ablation) | CAGE · CLEVRER composition | FID: 5.82 -> 4.98 | Helped |
| Set k=1 rather than k=3 (ablation) | CAGE · CLEVRER composition | CTRL: 1.615 -> 1.529 | Hurt |
| Set control sampling probability pi=0.9 rather than pi=0.1 (ablation) | CAGE · CLEVRER composition | CTRL: 1.615 -> 1.464 | Hurt |
| Set control sampling probability pi=0.5 rather than pi=0.1 (ablation) | CAGE · CLEVRER composition | CTRL: 1.615 -> 1.566 | No clear effect |
Surprising findings
- Controls from DINOv2 features generalize zero-shot to cross-domain objects (e.g., real objects transferred into CLEVRER scenes).
- On the BAIR optical-flow controllability score, CAGE performs slightly worse than the prior YODA model in the main setting.
Limitations
- Experiments limited to relatively small datasets, precluding direct comparison with large-scale video models.
- Controls are grid-aligned, limiting precision for shifts smaller than patch size.
- Irregular-shaped objects may split into multiple objects during cross-domain transfer.
Experimental setup
Datasets: CLEVRER BAIR EPIC-KITCHENS ROBONET
Benchmarks: CLEVRER BAIR EPIC-KITCHENS
Metrics: CTRL FID LPIPS PSNR SSIM FVD optical flow error
Baselines: YODA CADDY GLASS Huang et al. MAE features CLIP features
Builds on: YODA conditional flow matching DINOv2 VQGAN classifier-free guidance
Closest papers by meaning
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.90)
- DynaTokens: Teaching Dynamics to Camera-Controlled Video Models at Test Time (NeurIPS 2026 · similarity 0.88)
- Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation (AAAI 2024 · similarity 0.87)
- Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.87)
- UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics (CVPR 2025 · similarity 0.87)
- ONE-SHOT: Compositional Human-Environment Video Synthesis via Spatial-Decoupled Motion Injection and Hybrid Context Integration (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.