Research map / Video / Video-language understanding
PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PRISM, a 10,372-pair bilingual (English/Chinese) Manim instruction-to-code benchmark with funnel-style metrics (execution, spatial pass, PADVC, TD) and evaluates seven LLMs, finding a ~41-point Execution-Spatial Gap.
Problem
Existing benchmarks for code-based video generation are small and only check executability or coarse VLM judgments, missing spatial layout errors (overlap, out-of-bounds, leakage) across animation frames.
Approach
A deterministic, renderer-grounded spatial audit over full animation sequences, combined with a funnel from execution to spatial pass and prompt-aware dynamic complexity/temporal density diagnostics, applied to a large human-calibrated Manim benchmark.
Key results
- Execution-to-spatial pass drop averages about 41 percentage points (95% bootstrap CI [40.41, 42.11]) across seven LLMs.
- GPT-5.4 achieves the highest English execution rate at 0.945 but has high overlap (0.257) and out-of-bounds (0.518) rates.
- Gemini 3.1 Pro Preview leads on spatial pass with 0.572 (English) and 0.557 (Chinese).
Surprising findings
- Extended thinking mode does not reliably improve spatial organization; for GLM spatial shifts only about 1.3% despite roughly five extra minutes per sample.
- The highest execution-rate model (GPT-5.4) has the largest execution-to-spatial gap in both languages.
Limitations
- Evaluation uses a randomly sampled 20% subset of the benchmark for all models.
- PADVC and TD centering parameters are fitted on the benchmark's reference distribution, which may limit transfer to other settings.
- Instantiated only in Manim CE v0.19.0, with a single renderer.
Experimental setup
Datasets: PRISM
Benchmarks: PRISM
Metrics: execution success rate spatial pass rate PADVC TD render time overlap rate leakage rate out-of-bounds rate
Baselines: GPT-5.4 Gemini 3.1 Pro Preview Kimi K2 Thinking Claude Sonnet 4.6 Qwen3.5-397B-A17B GLM-5 DeepSeek-V3.2
Builds on: Manim
Closest papers by meaning
- Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation (NeurIPS 2026 · similarity 0.88)
- InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexity (CVPR 2026 · similarity 0.87)
- VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editing (ICML 2026 · similarity 0.86)
- Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation (CVPR 2024 · similarity 0.86)
- JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence (ICLR 2026 · similarity 0.86)
- Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflection (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.