Research map / 3D vision / 3D generation
Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile Gaussian Feature Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Feature4X distills features from 2D/video foundation models (SAM2, CLIP-LSeg, InternVideo2) into a compact scaffold-based 4D Gaussian feature field built from monocular video, and pairs it with GPT-4o agents for segmentation, editing, and VQA.
Problem
Extending 2D foundation model capabilities (segmentation, language-guided editing, VQA) to dynamic 3D/4D scenes reconstructed from casual monocular video, where multi-view data and per-frame annotations are scarce and existing static feature fields are memory-heavy and task-specific.
Approach
Distill multiple 2D foundation model features into one low-dimensional unified latent feature attached to MoSca motion-scaffold nodes and interpolated to dynamic Gaussians, decoded by lightweight task-specific MLPs, so new tasks need no full retraining; an LLM agent then drives editing and VQA through the feature field.
Key results
- Unified 32-dim feature field keeps PSNR at 25.197 vs MoSca 25.166 while reducing size to 95.457 MB from 593.907 MB for MoSca + Feature 3DGS.
- On DAVIS spatiotemporal VQA, global novel feature inference reaches 61.32% overall accuracy vs 49.06% for input video view.
- Feature-based inference runs about 3x faster (3.42 s vs 10.06 s) than rendering views and running the video VLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the full multi-task feature model instead of a single CLIP head (ablation) | Feature4X · Nvidia dataset | mIoU: 0.510 -> 0.503 | Hurt |
| use the full multi-task feature model instead of a single CLIP head (ablation) | Feature4X · Nvidia dataset | PSNR: 25.186 -> 25.197 | Helped |
| use the full multi-task feature model instead of a single CLIP head (ablation) | Feature4X · Nvidia dataset | accuracy: 0.880 -> 0.876 | Hurt |
| use Global Novel Feature inference instead of Input Video View | InternVideo2 VQA · DAVIS | Overall Acc.: 49.06 -> 61.32 | Helped |
| use Global Novel Feature inference instead of Input Video View | InternVideo2 VQA · DAVIS | Time (s): 10.02 -> 3.42 | Helped |
| use Local Novel Feature inference instead of Input Video View | InternVideo2 VQA · DAVIS | Spatial Acc.: 48.50 -> 54.50 | Helped |
| use Local Novel Feature inference instead of Input Video View | InternVideo2 VQA · DAVIS | Temporal Acc.: 49.84 -> 54.37 | Helped |
| use Local Novel View inference instead of Input Video View | InternVideo2 VQA · DAVIS | Overall Acc.: 49.06 -> 50.31 | Helped |
| use Global Novel View inference instead of Input Video View | InternVideo2 VQA · DAVIS | Overall Acc.: 49.06 -> 47.48 | Hurt |
| distill CLIP features with Feature4X instead of using MoSca + Feature 3DGS | Feature4X · Nvidia dataset | Size (MB): 593.907 -> 95.457 | Helped |
Surprising findings
- The naive MoSca + Feature 3DGS baseline gives slightly higher segmentation accuracy (0.881) than the proposed models (0.876-0.880).
Limitations
- Editing results are imperfect due to imperfect fine-grained feature alignment and non-optimal parameter tuning, as the paper admits.
- Evaluation is small-scale: quantitative results on one Nvidia scene set and 400 VQA questions over 50 DAVIS scenes; agentic editing is shown only qualitatively.
Experimental setup
Datasets: Nvidia dynamic scene dataset DAVIS
Benchmarks: Nvidia dataset DAVIS (400 VQA questions over 50 scenes)
Metrics: PSNR mIoU accuracy model size (MB) inference time (s)
Baselines: MoSca MoSca + Feature 3DGS InternVideo2 on input video view
Builds on: MoSca Feature 3DGS Dynamic 3D Gaussians SAM2 InternVideo2
Closest papers by meaning
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.88)
- 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.86)
- MVVBench: Benchmarking 4D Reasoning in Vision-Language Models (NeurIPS 2026 · similarity 0.86)
- 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities (NeurIPS 2024 · similarity 0.86)
- ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.