Research map / 3D vision / 3D generation
IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IGGT4D, a causal streaming feed-forward transformer (initialized from DA3-Giant) that jointly predicts camera pose, depth, ray maps and instance features, with a streaming codebook clustering for online 4D instance masks, plus the InsScene4D-147K dataset.
Problem
Online 4D scene understanding needs geometry and temporally consistent object identities from streaming video, but existing streaming reconstruction models are geometry-only and existing instance/semantic methods are offline, memory-hungry, or depend on external 2D cues.
Approach
A causal (frame-by-frame, KV-cached) geometry transformer with a Tri-DPT head whose instance branch is conditioned on depth and ray features via geometry-aware attention, paired with a lightweight streaming codebook clustering that replaces offline HDBSCAN, trained on a geometry-guided, projection-inherited instance annotation dataset.
Key results
- Camera pose AUC@3 average 0.4464 vs. best streaming baseline LingBot-Map 0.3060 across HiRoom, ETH3D, 7Scenes, ScanNet++.
- Instance tracking on long sequences: average T-mIoU 58.84 and T-SR 85.41, while IGGT runs out of memory.
- Streaming clustering keeps constant ~0.7 GB memory vs. IGGT's HDBSCAN reaching 25.85 GB at 16 frames and OOM at 32 frames.
Surprising findings
- Offline full-attention DA3 still beats the proposed causal streaming model on average camera pose AUC@3 (0.6046 vs 0.4464).
Limitations
- Supervised only; relies on large curated instance supervision and broader data coverage is needed for robustness and generalization.
- Evaluations are perception-centric; no embodied action or interaction feedback.
- Still trails offline full-attention DA3 on camera pose estimation in several settings.
Experimental setup
Datasets: InsScene4D-147K RealEstate10K ScanNet++ Aria Synthetic Environments SceneNet Infinigen Hypersim HOI4D Waymo Aria Digital Twin RoboTwin 2.0 Kubric
Benchmarks: HiRoom ETH3D 7Scenes ScanNet++ HOI4D Waymo PointOdyssey
Metrics: AUC@3 AUC@30 F1-score Chamfer Distance T-mIoU T-SR mIoU mAcc Match Rate
Baselines: VGGT MapAnything Pi3X DA3 CUT3R StreamVGGT Wint3R Stream3R LingBot-Map SpaTrackerV2+SAM SAM2 IGGT
Builds on: DA3 IGGT SAM2 VGGT CUT3R
Closest papers by meaning
- FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry (NeurIPS 2026 · similarity 0.89)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.89)
- Flux4D: Flow-based Unsupervised 4D Reconstruction (NeurIPS 2025 · similarity 0.88)
- EA3D: Online Open-World 3D Object Extraction from Streaming Videos (NeurIPS 2025 · similarity 0.88)
- X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-Modal Knowledge Transfer (AAAI 2024 · similarity 0.87)
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.