Research map / Video / Video segmentation and tracking
Can 4D Foundation Models Remember?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PersistBench, a reference-based benchmark built from 360° videos that evaluates object permanence, motion continuity, and appearance preservation of 12 4D foundation models, finding they remember far worse once objects leave the input view.
Problem
Existing 4D foundation models (reconstruction, camera-controllable video, and 360° generation models) are evaluated mainly with pixel-level metrics and lack ground truth for objects out of view, so their object-centric visual memory is unmeasured.
Approach
Use 360° videos as omniscient ground truth: build paired input (object visible then hidden) and reference (object always visible) perspective trajectories via constrained B-spline optimization, and score object permanence (SAM2 + VLM), motion continuity (centroid distance), and appearance (global DINOv2 masked pooling) against the reference.
Key results
- All 12 evaluated 4D foundation models score substantially lower on invisible segments than visible ones; e.g., GEN3C reaches 83.97% permanence on static invisible segments versus 95.96% visible.
- Explicit-conditioning models (GEN3C, TrajectoryCrafter, NeoVerse) generally show stronger invisible-segment memory.
- Argus achieves the highest static permanence (84.06%) and continuity (80.60%) among invisible-segment scores, while CubeComposer and Argus drop on dynamic objects.
Surprising findings
- Static memory scores are strongly correlated with dynamic memory scores across all metrics.
- 360° generation models that predict full panoramas do not inherently remember dynamic objects well, dropping sharply on the dynamic subset.
Limitations
- Scores depend on off-the-shelf components (ViPE poses, SAM2/SAM3 tracking, VLM judge) that may introduce errors, though rankings are largely preserved under replacement.
- Dataset drawn only from YouTube 360° videos, limiting scene diversity.
- Motion continuity is reference-based and ambiguous; rotation-dominant subset and camera-control perturbations affect rankings somewhat.
Experimental setup
Datasets: 360-1M
Benchmarks: PersistBench
Metrics: object permanence motion continuity appearance preservation
Baselines: CUT3R 4DGT CogNVS NeoVerse ReCamMaster TrajectoryCrafter GEN3C HyDRA ViewPoint Imagine360 Argus CubeComposer
Builds on: SAM2 SAM3 DINOv2 ViPE GeoCalib 360-1M
Closest papers by meaning
- Video World Models with Long-term Spatial Memory (NeurIPS 2025 · similarity 0.88)
- Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Video (CVPR 2025 · similarity 0.88)
- Probing the 3D Awareness of Visual Foundation Models (CVPR 2024 · similarity 0.87)
- Addressable Memory for Video World Models (NeurIPS 2026 · similarity 0.87)
- 4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models (CVPR 2026 · similarity 0.86)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.