Research map / Video / Video-language understanding

Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation

NeurIPS 2026 · Video › Video-language understanding · approach: Event and anomaly reasoning in video · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Animation2Code, a benchmark of 1,069 CodePen web animation videos paired with HTML/CSS/JS code, plus appearance (DreamSim+DTW) and temporal (tracklet-based) similarity metrics, and shows VLMs reproduce appearance but not temporal dynamics.

Problem

Whether VLMs can recover temporal dynamics from video and express them as executable code, since existing visual-to-code benchmarks target static images and existing video benchmarks are question-answering.

Approach

Frames animation video-to-code as de-rendering: a generated program is rendered and scored separately on appearance (DreamSim with DTW alignment over animated crops) and temporal similarity (CoTracker3 tracklet displacement correlation with Chamfer matching), validated against human preferences.

Key results

• GPT-5.4 with 2 FPS image frames achieves the highest appearance similarity (0.84) and 100.0% execution, while temporal similarity stays at most 0.31 (Gemini 3 Flash at 24 FPS).
• LoRA and full SFT of Qwen3-VL-8B raise execution to 98.6% and 94.9% but drop temporal similarity to 0.09 and 0.08.
• Iterative refinement of Qwen3-VL-8B lifts appearance from 0.70 to 0.73 and temporal from 0.24 to 0.28 after 3 iterations.

Surprising findings

• Finetuning Qwen3-VL-8B substantially degrades both appearance and temporal similarity despite improving execution success.
• Better appearance similarity does not consistently translate into better temporal alignment across models.

Limitations

• Temporal metric relies on CoTracker3, which is not optimized for synthetic web animations and can be noisy for small or fast elements.
• Dataset is limited to CodePen animations and excludes domains such as Manim, physics simulations, and specialized libraries.
• Human annotation temporal agreement is lower (alpha 0.73) than appearance agreement.

Experimental setup

Datasets: CodePen animation collections (1,069 pens)

Benchmarks: Animation2Code

Metrics: appearance similarity (DreamSim with DTW) temporal similarity (tracklet correlation, Chamfer) execution rate ROC-AUC vs human preference Krippendorff's alpha

Baselines: Gemini 3 Flash Preview Qwen3-VL-8B-Instruct GPT-5.4 Claude Sonnet 4.6 LLaMA 4 Scout SSIM CLIP AST-TSED VLM-as-a-judge (Gemini-3.1-Pro)

Builds on: DreamSim Motion Fidelity (Yatim et al.) CoTracker3 METAL Design2Code

Closest papers by meaning

• PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning (NeurIPS 2026 · similarity 0.88)
• VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models (ICLR 2026 · similarity 0.86)
• Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.86)
• Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification (CVPR 2025 · similarity 0.86)
• MotiF: Making Text Count in Image Animation with Motion Focal Loss (CVPR 2025 · similarity 0.85)
• Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs (NeurIPS 2024 · similarity 0.85)