Research map / Large language models / Language, safety and interpretability
The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAPO, a retrieval-augmented prompt optimization framework that rewrites user prompts for text-to-video models via a relation-graph word augmentation branch, a fine-tuned refactoring LLM, and a fine-tuned discriminator that selects between two candidate prompts.
Problem
Short or misaligned user prompts degrade text-to-video generation because T2V models perform best on prompts matching the training prompt distribution; prior LLM-based prompt optimization was designed for T2I and lacks guidance on vocabulary and sentence structure.
Approach
Build a relation graph of scenes and modifiers from training prompts, retrieve and merge relevant modifiers into the user prompt via an LLM, refactor the result with a LoRA-tuned LLM to match training prompt format, and use a fine-tuned discriminator LLM to pick between this and a directly rewritten prompt.
Key results
- On VBench, RAPO raises LaVie's total score from 80.89% to 82.38% and the multiple objects dimension from 37.71% to 64.86%.
- On VBench, RAPO raises Latte's multiple objects score from 29.55% to 52.78% and total score from 77.03% to 79.97%.
- On EvalCrafter, LaVie-RAPO reaches a final sum of 256 versus 248 for the naive LaVie prompts, and beats GPT-4 and Open-sora prompt rewriting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply RAPO prompt optimization | LaVie · VBench | Total Score: 80.89% -> 82.38% | Helped |
| apply RAPO prompt optimization | Latte · VBench | multiple objects: 29.55% -> 52.78% | Helped |
| apply RAPO prompt optimization | LaVie · VBench | multiple objects: 37.71% -> 64.86% | Helped |
| apply RAPO prompt optimization | Latte · VBench | Total Score: 77.03% -> 79.97% | Helped |
| apply RAPO prompt optimization | LaVie · EvalCrafter | Final Sum Score: 251 -> 256 | Helped |
| apply RAPO prompt optimization | Latte · EvalCrafter | Final Sum Score: 222 -> 227 | Helped |
| apply RAPO prompt optimization | LaVie · T2V-CompBench | object interactions: 0.760 -> 0.839 | Helped |
| apply RAPO prompt optimization | Latte · T2V-CompBench | object interactions: 0.792 -> 0.856 | Helped |
| add word augmentation (ablation) | RAPO on LaVie · VBench | Total Score: 80.37% -> 82.38% | Helped |
| add sentence refactoring (ablation) | RAPO on LaVie · VBench | Total Score: 79.75% -> 82.38% | Helped |
| add prompt selection (ablation) | RAPO on LaVie · VBench | Total Score: 81.58% -> 82.38% | Helped |
| replace GPT-4 with Mistral as L (ablation) | RAPO on LaVie · VBench | Total Score: 82.38% -> 82.25% | No clear effect |
Surprising findings
- Longer, more detailed prompts from GPT-4 and Open-sora often underperform the naive short prompts on several VBench dimensions, suggesting that over-detailed prompts can confuse T2V models.
Limitations
- Evaluated only on two T2V models (LaVie and Latte) in the main experiments.
- Discriminator labels depend on automatic evaluation metrics of generated videos rather than human judgments.
- Gains on some dimensions, such as spatial relationship for LaVie, are smaller or mixed relative to other methods.
Experimental setup
Datasets: Vimeo25M VBench prompts EvalCrafter prompts T2V-CompBench prompts
Benchmarks: VBench EvalCrafter T2V-CompBench
Metrics: VBench total score VBench dimension scores (multiple objects, object class, hum EvalCrafter final sum score T2V-CompBench attribute binding and object interaction score
Baselines: naive user prompt GPT-4 prompt rewriting Open-sora prompt refiner
Builds on: LaVie Latte VBench EvalCrafter T2V-CompBench
Closest papers by meaning
- PromptCoT: Align Prompt Distribution via Adapted Chain-of-Thought (CVPR 2024 · similarity 0.90)
- Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs (ICLR 2026 · similarity 0.89)
- Boosting Text-to-Video Generative Model with MLLMs Feedback (NeurIPS 2024 · similarity 0.89)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective (ICML 2025 · similarity 0.88)
- A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.