Research map / Video / Video generation
Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeuS-V, a neuro-symbolic metric that converts prompts to temporal logic specs, builds video automata from VLM confidences, and scores text-to-video alignment via probabilistic model checking, plus a 360-prompt temporal benchmark.
Problem
Existing text-to-video evaluation metrics emphasize visual quality and smoothness and miss temporal fidelity and text-to-video alignment, which matter for safety-critical uses.
Approach
Translate the prompt into propositions and a temporal logic specification (PULS), convert the generated video into a DTMC automaton using per-frame VLM semantic confidences, and compute the satisfaction probability with probabilistic model checking (Storm), calibrated by an empirical CDF.
Key results
- NeuS-V correlates with human text-to-video alignment annotations at r=0.479 to 0.624 across Gen-3, Pika, T2V-Turbo-v2 and CogVideoX-5B, versus VBench correlations between -0.087 and 0.089.
- Gen-3 achieves the highest overall NeuS-V score (0.723) on the prompt suite, versus 0.508 for Pika, 0.552 for T2V-Turbo-v2 and 0.589 for CogVideoX-5B.
- On MSR-VTT, NeuS-V separates aligned from misaligned captions with a 0.52 score difference versus 0.38 for VBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NeuS-V instead of VBench for text-to-video alignment evaluation | T2V model evaluations · human alignment annotations across Gen-3, Pika, T2V-Turbo-v2, and CogVideoX-5B | Pearson correlation: Gen-3: r = 0.089 vs r = 0.479; Pika: r = -0.087 vs r = 0.624; T2V-Turbo-v2: r = 0.010 vs r = 0.550; CogVideoX-5B: r = -0 | Helped |
| use NeuS-V rather than VBench to distinguish aligned from misaligned captions | video-caption alignment evaluation · MSR-VTT | aligned-versus-misaligned score difference: 0.38 vs 0.52 | Helped |
| replace temporal logic and formal verification with LLM-generated questions and VLM frame scoring (ablation) | NeuS-V · NeuS-V prompt suite | Pearson correlation with human alignment annotations: NeuS-V: r = 0.479, 0.624, 0.550, 0.540; GPT-4o + LLaMa-3.2-11B: r = 0.216, 0.296, 0.295, 0.264; GPT-4o + LLaVA-Video-7B: | Hurt |
| use 3 frames instead of 1 frame for VLM context (ablation) | NeuS-V · human alignment annotations across the four T2V models | Pearson correlation: Gen-3: 0.29 -> 0.48; Pika: 0.51 -> 0.62; T2V-Turbo-v2: 0.34 -> 0.55; CogVideoX-5B: 0.32 -> 0.54 | Helped |
| replace InternVL2-8B with LLaMa3.2-11B (ablation) | NeuS-V automaton construction · human alignment annotations across the four T2V models | Pearson correlation: Gen-3: 0.29 vs 0.15; Pika: 0.51 vs 0.16; T2V-Turbo-v2: 0.34 vs 0.08; CogVideoX-5B: 0.32 vs -0.02 | Hurt |
Surprising findings
- Replacing formal temporal logic and model checking with VLM-based VQA pipelines lowers correlation with human judgments, even when using the same VLMs.
- Current T2V models often rely on simple camera pans and zooms rather than true temporal interactions.
Limitations
- Lacks a mechanism to penalize unintended or extraneous elements in generated videos.
- Prompt suite is modest in size (360 prompts; 160 used for correlation).
- Relies on VLM calibration and confidence scores, and the VLM choice affects results (LLaMa3.2 overconfident).
Experimental setup
Datasets: MSR-VTT validation set NeuS-V prompt suite (360 prompts)
Benchmarks: NeuS-V prompt suite MSR-VTT
Metrics: Pearson correlation with human annotations satisfaction probability / NeuS-V score mean score difference between aligned and misaligned caption
Baselines: VBench GPT-4o + LLaMa-3.2-11B-Vision VQA pipeline GPT-4o + LLaVA-Video-7B VQA pipeline
Builds on: DSPy MIPROv2 Storm model checker VBench
Closest papers by meaning
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.89)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.89)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.89)
- Evaluation of Text-to-Video Generation Models: A Dynamics Perspective (NeurIPS 2024 · similarity 0.88)
- GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning (ICML 2025 · similarity 0.88)
- Can Text-to-Video Generation help Video-Language Alignment? (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.