Research map / Video / Video generation
Mimir: Improving Video Diffusion Models for Precise Text Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mimir fuses T5 encoder tokens with decoder-only Phi-3.5 LLM tokens (via normalization, zero-conv, and learnable semantic stabilizer tokens) inside a CogVideoX-5B diffusion transformer to improve text-to-video comprehension.
Problem
Video diffusion models rely on CLIP/T5 text encoders with limited text comprehension, while decoder-only LLMs offer better text understanding but their features distribution-mismatch and fluctuate, hindering direct use in T2V models.
Approach
A token fuser that harmonizes encoder and decoder-only LLM tokens: normalization plus learnable scale and zero-initialized convolution layers for non-destructive fusion, and learnable semantic stabilizer tokens paired with instruction tokens to stabilize fluctuating LLM features before feeding them into the diffusion transformer.
Key results
- Mimir achieves the best VBench scores across all reported dimensions, e.g., Multiple Objects 85.29% and Spatial Relationship 78.67%, versus 66.58%/68.25% for the CogVideoX-5B baseline in the ablation.
- In a blind user study with 10 participants, Mimir was preferred at 82.00% for instruction following, 83.65% for physics simulation, and 89.65% for visual quality.
- Naively adding decoder-only tokens to the encoder collapses training (Object Class 4.97%, Multiple Objects 0.00%); normalization and zero-conv fusion recover performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add decoder-only Phi-3.5 tokens using Mimir's token fuser | CogVideoX-5B · VBench | Multiple Objects: 65.70% -> 85.29% | Helped |
| Add decoder-only Phi-3.5 tokens using Mimir's token fuser | CogVideoX-5B · VBench | Spatial Relationship: 64.86% -> 78.67% | Helped |
| Directly combine encoder and decoder-only tokens (ablation) | CogVideoX-5B baseline · VBench | Multiple Objects: 66.58% -> 0.00% | Hurt |
| Add normalization to decoder-only tokens before fusion (ablation) | CogVideoX-5B with decoder-only tokens · VBench | Multiple Objects: 0.00% -> 65.24% | Helped |
| Add Semantic Stabilizer after normalization (ablation) | CogVideoX-5B with normalized decoder-only tokens · VBench | Multiple Objects: 65.24% -> 68.83% | Helped |
| Replace normalization-based fusion with ZeroConv fusion (ablation) | CogVideoX-5B with decoder-only tokens · VBench | Multiple Objects: 65.24% -> 84.98% | Helped |
| Add Semantic Stabilizer to ZeroConv fusion (ablation) | CogVideoX-5B with ZeroConv fusion · VBench | Multiple Objects: 84.98% -> 84.47% | No clear effect |
Surprising findings
- Directly summing decoder-only LLM tokens with encoder tokens causes training collapse, with near-zero VBench scores on several dimensions.
Limitations
- Experiments limited to a single LLM (Phi-3.5) and single diffusion backbone (CogVideoX-5B).
- Quantitative comparisons rely on VBench and a small blind user study of 20 prompts.
Experimental setup
Datasets: 500,000 curated video clips
Benchmarks: VBench user study (10 participants, 120 clips)
Metrics: Background Consistency Aesthetic Quality Imaging Quality Object Class Multiple Objects Color Spatial Relationship Temporal Style user preference rate for instruction following, physics simu
Baselines: ModelScopeT2V OpenSora OpenSoraPlan CogVideoX-2B CogVideoX-5B
Builds on: CogVideoX latent diffusion models LLM2Vec ControlNet-style zero convolution
Closest papers by meaning
- A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.90)
- Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation (CVPR 2025 · similarity 0.90)
- A Recipe for Scaling up Text-to-Video Generation with Text-free Videos (CVPR 2024 · similarity 0.90)
- Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.89)
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.89)
- Language Model Beats Diffusion - Tokenizer is key to visual generation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.