Research map / Video / Video generation
3DSPA: A 3D Semantic Point Autoencoder for Evaluating Video Realism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3DSPA, a 3D semantic point-track autoencoder (extending TRAJAN with depth and DINOv2 features) whose reconstruction score (Average Jaccard) serves as a reference-free realism metric for generated videos.
Problem
Automated, reference-free evaluation of generated video realism that captures semantics and 3D physical plausibility, beyond 2D temporal consistency metrics that need human annotation or paired real/fake data.
Approach
Train an autoencoder on 3D point trajectories (from CoTracker3 tracks lifted with VideoDepthAnything depth) enriched with DINOv2 features; the reconstruction error (AJ) of held-out query tracks is used as a realism score, since physically implausible motion is harder to reconstruct.
Key results
- On IntPhys2, 3DSPA reaches 76.92% (permanence, fixed) and 70.77% (solidity, fixed) win rates, outperforming listed VLMs and video models in most categories.
- On VideoPhy-2 physical commonsense, 3DSPA achieves Spearman 0.74 versus 0.76 for VideoPhy-2 AutoEval (fine-tuned) and 0.50 for the no-DINO ablation.
- On TAPVid-3D minival, 3DSPA averages 14.0 AJ, competitive with a fine-tuned CoTracker3 (13.5 AJ).
Surprising findings
- Adding DINOv2 semantic features matters more than 3D geometry for physical-violation detection: the no-3D variant performs comparably to full 3DSPA on IntPhys2.
- 3DSPA outperforms several fine-tuned VLM evaluators (VideoCon, VideoScore, VideoLLaVA) on VideoPhy-2 physical commonsense correlation.
Limitations
- Reconstruction and realism scores degrade when monocular depth (VideoDepthAnything) is noisy or ambiguous, propagating errors.
- Trajectories are not conditioned on past motion, limiting evaluation of long-term dynamics.
- Gains on IntPhys2 continuity are small or negative relative to baselines.
Experimental setup
Datasets: Kubric3D TAPVid-3D IntPhys2 VideoPhy-2 EvalCrafter
Benchmarks: TAPVid-3D minival IntPhys2 VideoPhy-2 EvalCrafter
Metrics: Average Jaccard APD Occlusion Accuracy win rate Spearman rank correlation
Baselines: TAPIR CoTracker BootsTAPIR CoTracker3 SpatialTracker SpatialTrackerV2 GPT-4o Qwen-VL 2.5 Gemini-1.5 Pro Gemini-2.5 Flash VideoMAEv2-g Cosmos-4B
Builds on: TRAJAN CoTracker3 VideoDepthAnything DINOv2 TAPVid-3D Perceiver
Closest papers by meaning
- Direct Motion Models for Assessing Generated Videos (ICML 2025 · similarity 0.88)
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.86)
- VideoPhy: Evaluating Physical Commonsense for Video Generation (ICLR 2025 · similarity 0.85)
- Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach (NeurIPS 2025 · similarity 0.85)
- GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (CVPR 2025 · similarity 0.85)
- Impossible Videos (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.