Research map / Video / Video generation

ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation

NeurIPS 2024 · Video › Video generation · approach: Compositional text-to-video diffusion · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ChronoMagic-Bench, a 1,649-prompt benchmark with two automatic metrics (MTScore for metamorphic amplitude, CHScore for temporal coherence) for evaluating time-lapse text-to-video generation, plus the 460k-clip ChronoMagic-Pro dataset.

Problem

Existing text-to-video benchmarks and metrics assess only visual quality and text relevance, overlooking metamorphic amplitude (degree of change over time) and temporal coherence in time-lapse videos.

Approach

Builds a time-lapse-focused benchmark across four categories (75 subcategories) and proposes a retrieval-based coarse metamorphic score (MTScore), a GPT-4o 5-point fine-grained score, and a point-tracking-based coherence score (CHScore), alongside a large time-lapse training set with captions generated by ShareGPT4Video.

Key results

• MagicTime (MTScore 0.479) and CogVideoX-2B (GPT4o-MTScore 3.09, MTScore 0.472 on ChronoMagic-Bench-150) are the only open-source models showing substantial metamorphic amplitude; most other models score low.
• Fine-tuning OpenSoraPlan v1.1 on 10k ChronoMagic-Pro clips with the Magic Training Strategy raised GPT4o-MTScore from 2.52 (baseline OpenSoraPlan v1.1 row, 2.19 in main table) to 3.05 and CHScore from 68.52 to 70.12.
• Human study with 212 participants over 16 prompts found MTScore, CHScore, and GPT4o-MTScore consistent with human perception, while UMTScore had the lowest Kendall and Spearman correlations.

Measured effects reported in the paper

Surprising findings

• UMTScore is lowest for MagicTime and CogVideoX-2B, the only models that follow time-lapse prompts, suggesting UMT-based text relevance is inconsistent with human perception.
• Closed-source DiT model Dream Machine leads on metamorphic amplitude, while ZeroScope obtains the best CHScore despite limited metamorphic amplitude.

Limitations

• Data sourced from YouTube with variable quality and CC BY 4.0 licensing restricting use to academic research
• A clear gap remains between automatic metrics (MTScore, CHScore) and human preferences
• Human study used only 16 prompts and six baselines per comparison

Experimental setup

Datasets: ChronoMagic-Bench ChronoMagic-Pro WebVid-10M Panda-70M ChronoMagic

Benchmarks: ChronoMagic-Bench ChronoMagic-Bench-150

Metrics: UMT-FVD UMTScore MTScore CHScore GPT4o-MTScore Kendall Spearman

Baselines: ModelScopeT2V ZeroScope T2V-zero LaVie AnimateDiff V3 VideoCrafter2 MagicTime CogVideoX-2B Gen-2 Pika-2.0 Dream Machine KeLing

Builds on: FETV VBench Panda-70M MagicTime Magic Training Strategy

Closest papers by meaning

• VideoPhy: Evaluating Physical Commonsense for Video Generation (ICLR 2025 · similarity 0.88)
• ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Way (CVPR 2025 · similarity 0.88)
• VideoMAR: Autoregressive Video Generation with Continuous Tokens (NeurIPS 2025 · similarity 0.87)
• Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification (CVPR 2025 · similarity 0.87)
• LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity (CVPR 2025 · similarity 0.87)
• PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation (CVPR 2025 · similarity 0.87)