Research map / Video / Video generation
EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EIDT-V, a model-agnostic, zero-shot, training-free text-to-video method that uses grid-based prompt switching in diffusion latent trajectories, with LLM-generated frame-wise prompts and CLIP-segmentation-based switch-time masks.
Problem
Generating coherent videos from image diffusion models without training or architectural modifications, balancing frame-to-frame coherence and variance across frames.
Approach
Switches prompts per grid cell at times set by CLIP-segmentation attention maps derived from LLM-identified differences between consecutive frame prompts, blending original and new diffusion trajectory latents so earlier switches add variance and later switches add coherence; works on latents only so it applies to any image diffusion model.
Key results
- EIDT-V SD1.5 reaches MS-SSIM 0.63 and temporal consistency 0.152, versus Free-Bloom's 0.672 MS-SSIM and 0.159 temporal consistency among SD1.5 baselines.
- EIDT-V SD3 Medium achieves MS-SSIM 0.81, LPIPS 0.184 and temporal consistency 0.087, the best numbers in Table 1.
- In an 8-participant user study, EIDT-V SD1.5 obtains the best overall mean ranking of 1.9 versus 2.3 for FreeBloom.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EIDT-V instead of Free-Bloom | SD1.5 video generation · Video generation prompts | LPIPS: 0.353 ± 0.082 -> 0.33 ± 0.1 | Helped |
| Use EIDT-V instead of Free-Bloom | SD1.5 video generation · Video generation prompts | MS-SSIM: 0.672 ± 0.095 -> 0.63 ± 0.137 | Hurt |
| Use EIDT-V instead of Free-Bloom | SD1.5 video generation · Video generation prompts | Temporal Consistency Loss: 0.159 ± 0.039 -> 0.152 ± 0.062 | Helped |
| Add Grid Prompt Switching to ChatGPT prompts (ablation) | EIDT-V with ChatGPT prompts · Ablation prompts | MS-SSIM: 0.132 ± 0.052 -> 0.588 ± 0.129 | Helped |
| Add Grid Prompt Switching to ChatGPT prompts (ablation) | EIDT-V with ChatGPT prompts · Ablation prompts | LPIPS: 0.723 ± 0.043 -> 0.384 ± 0.094 | Helped |
| Add Grid Prompt Switching to ChatGPT prompts (ablation) | EIDT-V with ChatGPT prompts · Ablation prompts | Temporal Consistency Loss: 0.389 ± 0.046 -> 0.157 ± 0.062 | Helped |
| Add Grid Prompt Switching to framewise prompts (ablation) | EIDT-V with framewise prompts · Ablation prompts | MS-SSIM: 0.124 ± 0.082 -> 0.63 ± 0.137 | Helped |
| Add Grid Prompt Switching to framewise prompts (ablation) | EIDT-V with framewise prompts · Ablation prompts | LPIPS: 0.686 ± 0.062 -> 0.33 ± 0.1 | Helped |
| Add Grid Prompt Switching to framewise prompts (ablation) | EIDT-V with framewise prompts · Ablation prompts | Temporal Consistency Loss: 0.403 ± 0.078 -> 0.152 ± 0.062 | Helped |
| Use framewise prompts instead of ChatGPT prompts (ablation) | EIDT-V without Grid Prompt Switching · Ablation prompts | MS-SSIM: 0.132 ± 0.052 -> 0.124 ± 0.082 | Hurt |
| Use framewise prompts instead of ChatGPT prompts (ablation) | EIDT-V without Grid Prompt Switching · Ablation prompts | LPIPS: 0.723 ± 0.043 -> 0.686 ± 0.062 | Helped |
| Use framewise prompts instead of ChatGPT prompts (ablation) | EIDT-V without Grid Prompt Switching · Ablation prompts | Temporal Consistency Loss: 0.389 ± 0.046 -> 0.403 ± 0.078 | Hurt |
Surprising findings
- Free-Bloom achieves a higher MS-SSIM (0.672) than EIDT-V SD1.5 (0.63), so the method is not best on every metric.
Limitations
- Occasionally replaces expected motion with minimal changes that do not convey natural movement.
- Artifacts such as limb elongation, especially with SD3; no configuration fully eliminates them.
- Late significant prompt changes may prevent refinement of fine details.
Experimental setup
Benchmarks: 50 self-made video generation prompts
Metrics: MS-SSIM LPIPS optical flow temporal consistency loss CLIP score user study ranking
Baselines: T2V-Zero DirecT2V Free-Bloom
Builds on: Text2Video-Zero DirecT2V Free-Bloom IP-Adapter Denoising diffusion implicit models
Closest papers by meaning
- Grid Diffusion Models for Text-to-Video Generation (CVPR 2024 · similarity 0.92)
- LLM-grounded Video Diffusion Models (ICLR 2024 · similarity 0.91)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.90)
- Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis (CVPR 2024 · similarity 0.90)
- ControlVideo: Training-free Controllable Text-to-video Generation (ICLR 2024 · similarity 0.90)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.