Research map / Video / Video generation
Boximator: Generating Rich and Controllable Motions for Video Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Boximator adds a frozen-backbone box-constraint control module (hard/soft boxes with self-tracking training) to video diffusion models PixelDance and ModelScope for fine-grained motion control.
Problem
Fine-grained object motion control in video synthesis is hard to specify with text or trajectories; existing box-based methods need text per box or lack soft constraints and training analysis.
Approach
A plug-in control module that encodes hard/soft bounding boxes with object IDs as tokens fed through an added self-attention layer in frozen video diffusion U-Nets, trained with a self-tracking technique (generating colored boxes in the video) and multi-stage hard-to-soft box curriculum.
Key results
- PixelDance + Boximator with box constraints improves FVD from 237 to 174 on MSR-VTT (ModelScope 239 to 216), with mAP rising from 0.094 to 0.349 on MSR-VTT.
- On ActivityNet, box constraints raise PixelDance mAP from 0.050 to 0.445 (about 8.9x).
- Human side-by-side study on 100 samples: motion control preferred for Boximator in 76.0% vs 2.2% for base model; video quality 35.2% vs 16.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add box constraints | PixelDance + Boximator · MSR-VTT | FVD: 237 -> 174 | Helped |
| add box constraints | ModelScope + Boximator · MSR-VTT | FVD: 239 -> 216 | Helped |
| add box constraints | PixelDance + Boximator · MSR-VTT | mAP: 0.094 -> 0.349 | Helped |
| add box constraints | PixelDance + Boximator · ActivityNet | mAP: 0.050 -> 0.445 | Helped |
| add box constraints | PixelDance + Boximator · UCF-101 | mAP: 0.060 -> 0.228 | Helped |
| remove self-tracking (ablation) | PixelDance + Boximator · MSR-VTT | mAP (Box): 0.349 -> 0.118 | Hurt |
| remove self-tracking (ablation) | PixelDance + Boximator · ActivityNet | mAP (Box): 0.445 -> 0.083 | Hurt |
| remove soft boxes (ablation) | PixelDance + Boximator · MSR-VTT | mAP (Box): 0.349 -> 0.235 | Hurt |
| remove soft boxes (ablation) | PixelDance + Boximator · ActivityNet | mAP (Box): 0.445 -> 0.248 | Hurt |
| train base model weights instead of freezing them (ablation) | PixelDance + Boximator · MSR-VTT | mAP: 0.349 -> 0.354 | No clear effect |
| train base model weights instead of freezing them (ablation) | PixelDance + Boximator · ActivityNet | mAP: 0.445 -> 0.404 | Hurt |
| add box constraints while conditioning on the first frame | PixelDance + Boximator · MSR-VTT | FVD: 113 -> 102 | Helped |
Surprising findings
- Without self-tracking, box alignment AP falls drastically (mAP 0.118 on MSR-VTT with box vs 0.349 with it), indicating direct loss optimization struggles to associate box tokens with objects.
- Training all U-Net weights gives no clear benefit over freezing the base model, and performs worse on ActivityNet.
Limitations
- AP metric depends on a noisy DINO+DEVA detector, so absolute scores are not success rates.
- CLIPSIM slightly drops when extra conditions (F0 or Box) are added.
- Evaluation on UCF-101 and ActivityNet uses automatically generated or manually annotated boxes of varying quality.
Experimental setup
Datasets: WebVid-10M MSR-VTT ActivityNet UCF-101 ActivityNet Entities
Benchmarks: MSR-VTT ActivityNet UCF-101
Metrics: FVD CLIPSIM mAP AP50 AP75
Baselines: PixelDance ModelScope FACTOR-traj Show-1 Phenaki LVDM MagicVideo
Builds on: GLIGEN NeRF DDIM FACTOR
Closest papers by meaning
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.90)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.89)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.88)
- Image Conductor: Precision Control for Interactive Video Synthesis (AAAI 2025 · similarity 0.88)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.87)
- Training-free Camera Control for Video Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.