Research map / Video / Video generation
TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TIV-Diffusion, an autoregressive diffusion model for text-driven image-to-video generation that uses slot-attention object-centric representations fused with text and Gumbel-Softmax slot selection to improve alignment and reduce object deformation.
Problem
Text-driven image-to-video generation (TI2V) struggles with misaligning text-described motion to the correct objects in the first frame and with object deformation or disappearance during movement.
Approach
Decompose the first frame into object slots via a pretrained Slot Attention encoder, align each slot with the text caption via cross-attention, and inject the text-enhanced slots into the diffusion U-Net through Gumbel-Softmax slot selection, alongside SPADE scale-offset modulation of fused text-image features.
Key results
- On CATER-GEN-v1, TIV-Diffusion reports SSIM 0.97, PSNR 37.15, LPIPS 0.06, FID 6.06, FVD 13.82, versus MAGE+ (PSNR 36.16, FID 6.41, FVD 21.15).
- On CATER-GEN-v2, TIV-Diffusion reports FID 6.67 and FVD 16.48 with PSNR 32.66, comparable to MAGE+ (PSNR 32.58) but with lower FID and LPIPS.
- On the Bridge dataset, TIV-Diffusion achieves SSIM 0.76, PSNR 17.85, LPIPS 0.24, FID 59.18, FVD 326.36, outperforming MAGE and Seer on all metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add object-centric representations (ablation) | TIV-Diffusion · CATER-GEN-v1 | FVD: 30.87 -> 13.82 | Helped |
| Add object-centric representations (ablation) | TIV-Diffusion · CATER-GEN-v2 | FID: 10.66 -> 6.67 | Helped |
| Add object-centric representations (ablation) | TIV-Diffusion · Double Moving MNIST | PSNR: 31.06 -> 32.99 | Helped |
| Add object-centric representations (ablation) | TIV-Diffusion · Modified Double Moving MNIST | FVD: 6.47 -> 4.03 | Helped |
| Replace SPADE modulation with cross-attention conditioning (ablation) | TIV-Diffusion · CATER-GEN-v2 | PSNR: 32.66 -> 29.23 | Hurt |
| Replace SPADE modulation with cross-attention conditioning (ablation) | TIV-Diffusion · CATER-GEN-v1 | PSNR: 37.15 -> 35.07 | Hurt |
| Replace the image encoder with a convolutional layer (ablation) | TIV-Diffusion · CATER-GEN-v1 | PSNR: 37.15 -> 37.05 | Hurt |
| Replace ConvGRU with a convolutional layer (ablation) | TIV-Diffusion · CATER-GEN-v1 | FVD: 13.82 -> 22.74 | Hurt |
| Use 3 slots instead of 5 (ablation) | TIV-Diffusion · CATER-GEN-v2 | PSNR: 32.66 -> 32.60 | Hurt |
| Increase denoising timesteps from T=250 to T=1600 (ablation) | TIV-Diffusion · CATER-GEN-v1 | PSNR: 36.79 -> 37.15 | Helped |
| Increase denoising timesteps from T=250 to T=1600 (ablation) | TIV-Diffusion · CATER-GEN-v1 | SSIM: 0.97 -> 0.97 | No clear effect |
| Use 11 slots instead of 5 (ablation) | TIV-Diffusion · CATER-GEN-v2 | PSNR: 32.66 -> 32.16 | Hurt |
Surprising findings
- On the Modified Double Moving MNIST, MAGE achieves higher SSIM (0.92) than TIV-Diffusion (0.91).
- Replacing SPADE modulation with cross-attention lowers CATER-v2 PSNR from 32.66 to 29.23.
Limitations
- More parameters than MAGE, increasing model size.
- Autoregressive strategy yields longer inference time.
- Generalization to unseen scenarios depends on training data quality and diversity.
Experimental setup
Datasets: Single Moving MNIST Double Moving MNIST Modified Double Moving MNIST CATER-GEN-v1 CATER-GEN-v2 Weizmann KTH Bridge
Benchmarks: Single Moving MNIST Double Moving MNIST Modified Double Moving MNIST CATER-GEN-v1 CATER-GEN-v2 Bridge
Metrics: SSIM PSNR LPIPS FID FVD
Baselines: TVP MAGE MAGE+ TiV-ODE Seer
Builds on: DDPM Slot Attention SPADE Gumbel-Softmax ConvGRU
Closest papers by meaning
- TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.89)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.89)
- Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search (NeurIPS 2025 · similarity 0.89)
- MotiF: Making Text Count in Image Animation with Motion Focal Loss (CVPR 2025 · similarity 0.89)
- VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.