Research map / Architectures and efficiency / Attention and transformers

FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Block-sparse attention kernels · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FrameVGGT, an inference-only bounded-memory scheme for streaming geometry transformers that keeps frame-wise KV segments as coherent units (selected by key-space segment dissimilarity) plus a sparse anchor tier, evaluated on 3D reconstruction, video depth and pose.

Problem

Streaming visual geometry transformers accumulate an unbounded KV-cache over long streams, and under fixed memory budgets token-level cache retention fragments the geometric context needed for stable multi-view matching.

Approach

Shift bounded cache retention from token granularity to frame-wise KV segments summarized by a mean key prototype; select diverse segments via greedy farthest-first over cosine distance, and add a sparse anchor tier for persistent long-range references. Motivated by a directional-concentration diagnostic and variance bound.

Key results

• On NRGBD, Ours(16) reaches Acc 0.053 and Comp 0.030 versus 0.078/0.035 for InfiniteVGGT, the strongest baseline.
• On TUM pose estimation, Ours(24) reports ATE 0.0385 versus 0.0478 for InfiniteVGGT.
• Memory footprint reported as 1.9-3.7 GB for 12-24 blocks versus 6.9 GB for InfiniteVGGT and 10.3 GB for XStreamVGGT.

Surprising findings

• Recent-K sliding-window buffers consistently degrade reconstruction, depth, and pose, suggesting adjacent frames are over-allocated.

Limitations

• Fixed inference-time policy with no training or learned memory scoring.
• Anchor thresholds are fixed across datasets and the design space of temporal memory allocation is not exhausted.
• Diagnostic proxies (context-damage, geometric-context score) are conceptual and not formally validated.

Experimental setup

Datasets: 7-Scenes NRGBD Long3D Bonn TUM-dynamics

Benchmarks: 7-Scenes NRGBD Long3D Bonn TUM-dynamics

Metrics: Acc Comp NC Abs Rel delta<1.25 ATE RPE trans RPE rot

Baselines: CUT3R Point3R TTT3R XStreamVGGT InfiniteVGGT OVGGT

Builds on: StreamVGGT InfiniteVGGT CUT3R

Closest papers by meaning

• GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction (NeurIPS 2026 · similarity 0.90)
• LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging (CVPR 2026 · similarity 0.89)
• IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer (NeurIPS 2026 · similarity 0.89)
• Rethinking the State Update Gate for Long-Sequence Recurrent 3D Reconstruction (NeurIPS 2026 · similarity 0.88)
• TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention (NeurIPS 2026 · similarity 0.87)
• V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models (NeurIPS 2026 · similarity 0.86)